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Preface 



For the thirty-third year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Jacksonville, FL. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes 1 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Editor 
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Instructors’ and Learners’ Attitudes Toward e-learning within a College of Education 

In the past two decades, there have been a wide variety of web-based tools that have given rise to electronic 
learning (e-learning) worldwide. E-learning opens new possibilities to learners and encourages educational 
innovations. E-leaming also provides a wide range of opportunities for students who may not have previously had 
the chance to participate in a higher-learning program. Because e-learning is trumping the barriers of location and 
time, e-learning is opening new doors to students who may already be working, who may have disabilities, or who 
may not be classified as a traditional student in one way or another (Chung, 2008). 

The morphing demographics of the typical learner and the changing approaches to education on a large 
scale complement each other. According to Allen and Seaman, “81% of all institutions of higher education in the 
United States offer at least one fully online or blended course, and 67% recognized online education as a critical 
long-term strategy for their institution. In the United States, the enrollment of online learners grew to approximately 
3.5 million, a 21% increase since 2002,” (as cited in Cleveland-Innes & Garrison, 2010, p. 1 68). When learners use a 
web-based learning environment, they can conquer the limitation of space and time to establish a convenient 
learning environment (Chen, 2009). Zaharias and Poylymenakou (2009) state that educational institutions have been 
implementing information technologies to improve education and teaching significantly during the last two decades. 
They identified e-learning as an enabler for people in different organizations to keep up with the latest changes that 
are happening in the business world. 

Khan (2005) states that e-learning encompasses web-based instruction (WBI), online learning (OL), mobile 
learning (m-learning), and web-based learning (WBL). He states, “E-learning can be viewed as an innovative 
approach for delivering well-designed, learner-centered, interactive, and facilitated learning environments to anyone, 
anyplace, anytime, by utilizing the attributes and resources of various digital technologies along with other forms of 
learning materials suited for open, flexible, and distributed learning environments” (p. 3). 

Because of the many new options available to e-leamers, students have the potential to be more selective 
about the quality of their e-learning experience. The decision of the student to continue with his or her distance- 
learning program or for the instructor to continue teaching online depends on several factors. A strong indicator that 
the learner will continue with his or her e-learning program is his or her success. The student can experience success 
in several ways, one being success with an ease of managing the technology of the program, and another being 
success in understanding and applying the content information of the course. 

It is evident from previous research that, aside from the content that the student will study, the design of the 
e-learning platform is crucial to the students’ success (Liaw, 2008; Chung, 2008; Lu, 2010). The design of the 
program includes not only the technological components, but the interactive components as well. Concerns about 
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computers, the instructor’s attitude toward e-learning, the flexibility of e-learning course, the quality of e-learning 
course, its apparent usefulness, the apparent easiness of use, and the variety in assessments are critical factors 
affecting learner satisfaction (Liaw, 2008). Students want a program that is easily navigable, and one in which they 
can have access to feedback from the instructor and other students. E-learning is changing the way in which 
education is taught, requiring more interaction on the part of the student as well as the instructor (Chung, 2008). 

This interaction is an especially important component of e-learning, because it motivates students through additional 
support and opportunities for deeper understanding. If the format of an e-learning program is easy to use and makes 
the student feel successful, the student will most likely have a positive attitude toward continued e-learning. All of 
these components contribute to increased student satisfaction, which encourages the learner to continue with his or 
her e-learning program. 

Other factors influence a student’s continuation of e-learning as well. Self-efficacy, gender, learning style, 
and job status all impact a student’s attitude and decisions about e-learning (Lu, 2010). Furthermore, physical and 
psychological factors can either encourage or hinder students’ attitude and success in an e-learning setting 
(Zandvliet, 2003). Recent studies suggest that including a range of social factors in e-learning, such as what students 
and instructors believe about learning, should be considered rather than focusing mostly on technology-based tools 
(Bielaczyc. 2006). From Triandis’ (1971) point of view, attitudes consist of three different components: affective, 
cognitive, and behavioral. The affective component includes statements of likes and dislikes about certain objects. 
The cognitive part refers to statements of a student or instructor that provides rational for the value of an object. The 
behavioral aspect explains what a student or instructor actually does or intends to do. 

All of these elements of e-learning help form a students’ general attitude toward e-learning, which greatly 
affects his/her decision to either continue or terminate an e-learning program. That is why it is so important for 
designers to be strategic in constructing technology that is easy to use, as well as create an overall experience that is 
supportive and builds upon the student’s successes. Knowing students’ attitudes and assumptions toward e-learning, 
and researching critical factors that affect students’ behaviors toward e-learning can help in designing a more 
effective course that will encourage success (Chang, 2008). According to Liaw’s research,“learner computer 
anxiety, instructor attitude toward e-learning, e-learning course flexibility, e-learning course quality, perceived 
usefulness, perceived ease of use, and diversity in assessments are the critical factors affecting learners’ perceived 
satisfaction” (2008, p. 864). 

Liaw, Fluang, and Chen (2007) claim that only a small amount of e-learning literature assesses both 
instructor’ and student’ attitudes toward using e-learning as a teaching and learning tool. Because individual student 
attitudes play such an important role in determining whether or not they will chose to continue with their e-learning 
program, it is important to continue research on this topic. 

E-learning is defined by Liaw, Fluang, and Chen (2007) as “the use of Internet technologies to deliver a 
broad array of solutions that enhance knowledge and performance” (p. 1067). According to Rosenberg (2001), e- 
learning is based on three essential criteria: it is networked, it is delivered to the end-user via a computer using 
Internet technology, and it focuses on the broadest view of learning. E-learning is learner-centered, uses multiple 
instructional techniques, and provides opportunities for collaboration. 

Liaw (2004) describes three considerations for designing effective e-learning environments: learner 
characteristics, instructional structure, and interaction. Instruction in e-learning must be tightly structured and highly 
managed (Fulkerth, 1997). Some advanced instructional strategies such as cooperative learning and problem solving 
are widely implemented in e-learning environments. According to Spiro, Feltovich, Jacobson, and Coulson (1995), 
multimedia instruction helps learners to develop complex cognitive skills. E-learning environments offer 
opportunities for group interaction among learners and between learners and instructors. Group interaction is 
essential in applying cooperative learning in e-learning. This environment helps learners to construct their 
knowledge through their interaction with their peers (Vygotsky, 1978). 

In addition, learner analysis is essential in order to get a clear understanding of the target population and 
provide instruction that meets their needs. Learner characteristics that need to be identified include attitudes, 
motivation, beliefs, and confidence (Passerini & Granger, 2000). E-learning emphasizes autonomous learning 
environments that provide opportunity for self-directed learning. Therefore, it is imperative that learners have time 
management skills and are self-starters. 

Knowing learners’ attitudes regarding e-learning has taken the attention of information system research 
(Bishop, 2006). Smith, Caputi and Rawstorne (2000) state that “computer attitude is defined as a person's general 
evaluation or feeling of favourableness or unfavourableness toward computer technologies (i.e. attitude towards 
objects) and specific computer-related activities (i.e. attitude towards behaviors)” (p. 61). 

A purpose for studying learners’ attitudes toward e-learning is that they might reflect the learners’ 
subsequent use of the technology (Smith, Caputi & Rawstorne, 2000). As reported by Bishop (2006), there are other 
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factors that could affect the learners’ attitude such as goals, beliefs, or values. For example, if participating in a 
certain computer activity meets the learners’ goals and needs, this might increase the learners’ positive attitude 
toward computer usage. The interaction between the learner and the e-learning context is defined by hierarchical 
satisfaction. Subsequently, the learners who feel secure and meet his or her primary and higher needs would have a 
better attitude towards the process and it would increase his or her e-learning participation. 

On the other hand, the learner who doesn’t have his or her lower or primary needs met would likely not 
participate in e-learning activities because of the negative attitude towards the technology (Bishop, 2006). When 
learners have either positive or negative attitudes towards a new technology, those attitudes directly affect behavior 
and therefore the use of the technology, no matter the technology’s level of advance (Liaw, Huang, & Chen, 2007). 

There is often a connection between learners’ attitudes and their computer usage experience. Two aspects 
of computer experience can be seen to directly affect the learners’ attitude. The first one is the subjective experience, 
which relates to the feelings and thoughts of the learners toward their computer usage. The second is the objective 
experience, which relates to individual computer interaction (Smith, Caputi, & Rawstorne, 2000). 

Sometimes the e-learning context will affect the learners’ already established attitude towards computer 
use. When e-learning is used with weak instruction, that can frustrate or lose the learners’ attention. Learners will 
have a better attitude regarding e-learning when the instructors meet the learners’ social, personal, and psychological 
needs. Social characteristics include the learners’ preferences or attitude towards group or individual work. Personal 
characteristics are the learners’ beliefs, attitudes, anxieties, and motivations. Psychological characteristics are things 
like the learners’ age or gender (Liaw, 2004). 

Another way of seeing this idea is addressed by Federico (2000). In his study on learning styles and 
network-based instruction he found, “students with assimilating and accommodating learning styles demonstrated 
significantly more agreeable attitudes toward varied aspects of network-based instruction than students with 
converging and diverging learning styles” (Federico, 2000, p. 359). What this means for learners using e-learning is 
that those who are more adept at abstract conceptualization will produce a more favorable attitude than individuals 
who prefer concrete experiences. An explanation for this is provided by Liaw (2004), stating that e-learning is a 
more independent and autonomous environment. Subsequently, it would stand to reason that an abstract learner 
would find this appealing while a concrete one would not, since the concrete learner would be more favorable to a 
structured, traditional learning environment. 

Jones and Issroff (2005) argued that social emotional issues must also be considered when studying e- 
learning. They suggested that the technology must be safe for learning or, in the case of on-line communities safe for 
disclosure to be rewarding for the learners. This means that if the learner does not feel secure with using the 
technology, they will not trust it and therefore not be able to develop or divulge their emotions in an e-learning 
capacity. This would hinder them reaching any emotional goals that they were using e-learning to fulfill and so the 
exercise of using the technology would be useless. 

According to Crook (2000), education in both a traditional and e-learning setting is far more effective when 
collaboration between students is used, such as in computer-mediated conferencing or on-line communication. His 
main point is that to be collaborative one must be motivated to interact and cooperate. Subsequently, for e-learning 
to be effective, the learner must be motivated to use it and motivated to collaborate with his or her peers and 
teachers to use the technology effectively. 

Tseng, Chiang, and Hsu (2008) studied student satisfaction in using e-learning to solve problems and found 
that the satisfaction was much lower at the beginning because the learners had to be taught the new way of learning 
while trying to solve the problem at the same time. However, when teaching assistants were introduced to help the 
learners with understanding the new e-learning the satisfaction went up. They believed this was due to the learners 
being able to collaborate and discuss their problems with the teaching assistants. This shows that e-learning can 
become a useful resource even to those who dislike it at first, as long as the reasons they dislike it are addressed. 

With so many new options becoming available to learners, e-learning is going to continue to grow. 

Learners who are reluctant to try this technology are going to find themselves at a disadvantage. By addressing some 
of the factors that affect a learners’ desire to use e-learning, even those that prefer traditional instruction can be 
made comfortable and capable in regards to this new trend. 

The current study investigated instructors’ and learners’ attitudes toward e-learning. For both groups, 
reports of technology experience and attitudes toward e-learning were collected. In addition to viewing these 
independently, the relationship between reported technology experience and attitudes toward e-learning were 
examined. Finally, it was possible to compare some of the instructor and learner attitudes. 
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Methodology 



In this study, a sample of instructors and a sample of students were given similar surveys regarding their 
attitudes towards e-leaming. Both groups were selected from the same institution and the data for the two groups 
was collected at the same time. The specific method used for each group is described in a separate section below. 

Instructor attitudes 

Participants. The instructor participants were drawn from the college of education/behavioral sciences 
faculty at a medium-sized, public university in the Western United States. With the permission of the college’s dean, 
an email was sent to the full-time faculty of the college (approximately 1 10 individuals) seeking participants. Of 
these, 37 instructors participated by completing the online survey. 

Instrument. The questionnaire for the instructor participants was a slightly modified version of an 
instalment created by previous researchers (Liaw, Huang, & Chen, 2007). It included: (a) one question regarding the 
number of e-learning courses taught (with four possible responses), (b) six questions asking them to rate their 
experience in using various technologies (with a 7-point Likert response, ranging from no experience to well 
experienced), and (c) 19 statements regarding attitudes toward e-learning to which they were to respond (using a 7- 
point Likert response, ranging from strongly disagree to strongly agree). The last 19 items represented six subscales: 
perceived self-efficacy (three items), perceived enjoyment (three items), perceived usefulness (three items), 
behavioral intention to use e-learning (three items), perceived system satisfaction (three items), and multimedia 
instruction (four items). 

Procedure. The full-time instructors within the college were sent an email invitation to participate in the 
research. This email message contained an informed consent disclosure and a link to the on-line version of the 
survey for those wishing to participate. The survey was available for three weeks. At the beginning of the last week, 
a second call to participate was again sent to the full list. After three weeks, the survey was closed and the data were 
downloaded for analysis using the statistical software SPSS. 

Variables. The independent variables were the instructor-reported computer use and experience. The 
experience items were examined individually and as a composite measure. 

The dependent variables were instructor attitudes. These were examined individually and according to the 
six subscales: perceived self-efficacy, perceived enjoyment, perceived usefulness, perceived system satisfaction, 
behavioral intention to use e-learning and multimedia instruction. 

Learner attitudes 

Participants. The learner participants were drawn from the same medium-sized, public university in the 
Western United States. Because of policies regarding access to student e-mail addresses and solicitations via email, 
learner participants were approached to complete pencil-and-paper surveys at several campus locations near the 
college of education/behavioral sciences. Data collection ceased when 105 surveys had been completed. 

Instrument. The questionnaire for the learner participants was also a slightly modified version of an 
instalment created by previous researchers (Liaw, Huang, & Chen, 2007). It included: (a) one question regarding the 
number of e-learning courses in which they had participated (with four possible responses), (b) four questions asking 
them to rate their experience in using various technologies (with a 7-point Likert response, ranging from no 
experience to well experienced), and (c) 15 statements regarding attitudes toward e-learning to which they were to 
respond (using a 7-point Likert response, ranging from strongly disagree to strongly agree). The last 15 items 
represented three subscales: e-learning as a self-paced learning environment (six items), e-learning as an effective 
learning environment (three items), e-learning as a multimedia environment (three items), and teachers as an 
instaictor-led learning environment (three items). 

Procedure. In order to solicit participation, students were approached in public areas on campus with as 
little disaiption as possible. They were asked to volunteer to complete a short survey on e-learning. Those choosing 
not to respond were thanked. Participants were given a written introduction on the study including an informed 
consent form and the actual questionnaire. Upon completion, it was immediately returned to the researchers. A total 
of 105 learner participants were recanted during the one-week of data collection. The data were entered into the 
statistical software SPSS for analysis. 

Variables. The independent variables were the learner-reported computer use and experience. The 
experience items were examined individually and as a composite measure. 
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The dependent variables were learner attitudes. These were examined individually and according to the 
four subscales: e-learning as a self-paced learning environment, e-learning as an effective learning environment, e- 
learning as a multimedia instruction environment, and teachers as an instructor-led learning environment. 

Results 



Instructor experience and attitude 

Table 1 shows the instructors’ reported experience in using e-learning environments. 



Table 1 



Instructors’ experience in using e-learning environments 





Frequency 


Percent 


No courses 


9 


24.3 


1-2 courses 


10 


27.0 


3-4 courses 


4 


10.8 


More than 4 courses 


14 


37.8 


Total 


37 


100.0 



Descriptive statistics of instructors’ computer use and experience are shown in Table 2. The strongest 
reported experience was in using the Internet (M=6.51, SD= 0.69). However, they weakest reported experience was 
in e-learning (M= 4.97, SD= 1.80). 

Table 2 



Instructors’ experience in using technologies 





N 


M (SD) 


Experience using operating systems 


37 


5.03 


(1.89) 


Experience using the Internet 


37 


6.51 


(0.69) 


Experience using word processing packages 


37 


6.41 


(0.86) 


Experience using PowerPoint 


37 


6.08 


(1.19) 


Experience using computers as a teaching assisted tool 


37 


5.41 


(1.59) 


Experience using e-learning 


37 


4.97 


(1.80) 


Overall Experience with Learning Technologies 


37 


5.73 (1.03) 



Note: Using a 7-point scale with 1= “no experience” to 7 = “well experienced.” 



Table 3 presents the instructors’ attitudes towards e-learning. The data show that instructors had a positive 
overall attitude toward e-learning environments (M= 4.97, SD= 1.36). The highest reported attitude subscale was 
perceived self-efficacy (M= 5.56, SD=1.26), while the lowest reported attitude was multimedia instruction (M= 
4.31, SD= 1.41). 

Table 3 



Instructors’ attitudes towards e-learning 





N 


M (SD) 


Perceived self-efficacy 


37 


5.56 (1.26) 


I feel confident making online instruction 


37 


5.03 (1.80) 


I feel confident using the Internet 


37 


6.43 (0.87) 


1 feel confident using e-learning environments 


37 


5.22 (1.63) 


Perceived enjoyment 


37 


4.98 (1.67) 


I enjoy using computers as a teaching assisted tool 


37 


5.62 (1.62) 


I enjoy using e-learning environment for teaching purpose 


37 


4.62 (2.06) 
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I enjoy using online instruction for teaching 


37 


4.70 


(1.91) 


Perceived usefulness 


36 


4.96 


(1.81) 


I believe using e-learning environments is helpful for learning 


36 


5.03 


(1.81) 


1 believe using e-learning environments is helpful for teaching 


37 


4.86 


(1.84) 


I believe using online instruction is useful for teaching 


37 


4.95 


(1.79) 


Behavioral intention to use e-learning 


37 


5.44 


(1.50) 


1 intend to use e-learning to assist my teaching 


37 


5.27 


(1.71) 


1 intend to use online instruction to assist my teaching 


37 


5.16 


(2.06) 


I intend to use the Internet to assist my teaching 


37 


5.89 


(1.45) 


Perceived system satisfaction 


36 


4.53 


(1.61) 


1 am satisfied with using e-learning environments 


36 


4.22 


(1.87) 


1 am satisfied with using MS-Word, MS-PowerPoint files as multimedia 








instruction 


37 


4.84 


(1.82) 


I am satisfied with using online instruction 


37 


4.43 


(1.77) 


Multimedia instruction 


35 


4.31 


(1.41) 


I like to use voice media instruction 


37 


3.35 


(1.99) 


1 like to use image media instruction 


35 


5.17 


(1.81) 


I like to use animation media instruction 


37 


3.86 


(1.83) 


I like to use colorful text media instruction 


36 


4.67 


(1.82) 


Overall Attitude 


33 


4.97 (1.36) 



Note: Using a 7-point scale with 1= “strongly disagree” to 7 = “strongly agree.” The overall reliability of the attitude 
scale using Cronbach’s alpha was 0.96. The reliabilities for individual subscales were: perceived self-efficacy 
(0.80), perceived enjoyment (0.87), perceived usefulness (0.99), behavioral intention to use e-learning (0.81), 
perceived system satisfaction (0.86), and multimedia instruction (0.76) 

Each of the six attitude subscales was then correlated with the instructor’s overall experience with learning 
technologies (see Table 4). All six correlations were statistically significant. The strongest correlations were 
between overall technology experience and perceived self-efficacy (r = .78) and perceived enjoyment (r = .74). 

Table 4 



Correlation of overall technology experience with the attitude subscales 





N 


r 


Perceived self-efficacy 


37 


* 

.78 


Perceived enjoyment 


37 


.74“ 


Perceived usefulness 


36 


.49“ 


Behavioral intention to use e-learning 


37 


.62“ 


Perceived system satisfaction 


36 


.57“ 


Multimedia instruction 


35 


.67“ 



** Correlation is significant at the 0.01 level (2-tailed). 
* Correlation is significant at the 0.05 level (2-tailed). 



Results of learners’ attitudes 



Table 5 shows the learners’ reported experience in using e-learning environments. 




Table 5 



Learners’ experience in using e-learning environments 





Frequency 


Percent 


No courses 


38 


38.8 


1-2 courses 


30 


30.6 


3-4 courses 


17 


17.3 


More than 4 courses 


13 


13.3 


Total 


98 


100.0 



Descriptive statistics learners’ computer use and experience are shown in Table 6. The strongest reported 
experience was in using email (M=6.45, SD= 1.01). The weakest reported experience was in coding web pages (M= 
2.96, SD= 1.98). 



Table 6 



Learners’ experience in using technologies 





N 


M (SD) 


Experience using Web browsers 


105 


5.94 (1.29) 


Experience using e-mail 


105 


6.45 (1.01) 


Experience using word processing packages 


105 


5.97 (1.29) 


Experience coding web pages 


104 


2.96 (1.98) 


Overall Exp. with Learning Technologies 


104 


5.33 (0.99) 



Note: Using a 7-point scale with 1= “no experience” to 7 = “well experienced.” 



Table 7 presents the learners’ attitudes towards e-learning. The data show that students had a positive 
overall attitude toward e-learning environments (M= 4.70, SD= 0.98). The highest reported attitude subscale was e- 
learning as a multimedia instruction environment (M= 5.46, SD=1.27), while the lowest reported attitude was 
toward e-learning as an effective learning environment (M= 4. 14, SD= 1.33). 

Table 7 



Learners’ attitudes towards e-learning 





N 


M (SD) 


E-learning as a self-paced learning environment 


97 


4.34(1.14) 


I can learn actively in the e-learning environment 


105 


4.35 (1.57) 


I have more opportunities to create my own knowledge in the e-learning 
environment 


104 


4.42 (1.39) 


The hyper text online instruction can enhance my learning motivation 


104 


3.60 (1.48) 


I can discuss actively with others in the e-learning environment 


103 


4.00 (1.65) 


I can read the online instruction actively 


102 


4.96(1.51) 


I can find information actively in the e-learning environment 


104 


4.85 (1.42) 


E-learning as an effective learning environment 


98 


4.14(1.33) 


The e-leaming environment improves my thinking skills 


104 


3.96 (1.39) 


The e-learning environment enhances my problem-solving skills 


103 


4.14(1.50) 


The e-learning environment provides various aspects to solve problems 


101 


4.24 (1.48) 


E-learning as a multimedia instruction environment 


101 


5.46 (1.27) 


I like colorful pictures in online instruction 


105 


5.48 (1.39) 
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I like learning videos in online instruction 


103 


5.56 (1.58) 


I like the animated online instruction 


102 


5.33 (1.56) 


E-learning as an instructor-led learning environment 


105 


5.00 (1.36) 


I like the instructor's help and suggestions in the e-learning environment 


105 


5.16(1.64) 


I like the instructor's voice and image in the e-learning environment 


105 


4.83 (1.55) 


I like the instructor's online multimedia instruction in the e-learning 






environment 


105 


5.02 (1.5) 


Overall Attitude 


87 


4.71(0.98) 



Note: Using a 7-point scale with 1= “strongly disagree” to 7 = “strongly agree.” The overall reliability of the 
attitude scale using Cronbach’s alpha was (0.91). The reliabilities for individual subscales were: self-paced learning 
environment (0.85), effective learning environment (0.92), multimedia instruction environment (0.78), and 
instructor-led learning environment (0.84). 

Each of the four attitude subscales was then correlated with the learner’s overall experience with learning 
technologies (see Table 8). The correlations between overall technology experience and e-learning as a self-paced 
learning environment (r = 0.28) and e-learning as an instructor-led learning environment (r = 0.21) were statistically 
significant. However, both were extremely weak. 



Table 10 

Correlation of overall technology experience with the attitude subscales 

N r 

E-learning as a self-paced learning environment 96 .28 

E-learning as an effective learning environment 97 .18 

E-learning as a multimedia instruction environment 100 .10 

E-learning as an instructor-led learning environment 104 .21 

**. Correlation is significant at the 0.01 level (2-tailed). 

*. Correlation is significant at the 0.05 level (2-tailed). 



Discussion 

Overall, the instructors in this study had a positive attitude toward e-learning. When comparing the six 
instructor’s attitudes subscales of the instructors’ attitudes, perceived self-efficacy was rated the highest. The 
instructors felt very confident using the Internet, but also expressed confidence in online instruction and in using e- 
learning environments. Yuen & Ma (2008) found similar results in their study that self-efficacy and perceived ease 
of use influenced instructors’ intention to use the e-learning environment. The behavioral intention to use e-learning 
was the second highest rated subscale. In general, the instructors reported an intention to use e-learning in the future. 
In terms of actual media usage, the instructors reported a stronger preference for using images and colorful text. On 
average, they reported a dislike for using animation and voice media. It is not clear from this research whether this is 
due to the perceived effort and time required of different media or the instructors’ attitudes towards the effectiveness 
of the various types. In general, the instructors believed that the e-learning environment was helpful for learning and 
teaching and they enjoyed using computers as a teaching tool. 

There was a positive relationship between the instructors’ experience with technology and their attitudes 
toward e-learning. In this study, self-efficacy had the strongest relationship with instructors’ experiences. Instructors 
who were experienced in using technology felt more confident in their abilities in using an e-learning environment. 
They also were more likely to enjoy using e-learning environments. 

The learners in the study also had a positive attitude toward e-learning. They reported the highest attitudes 
towards e-learning as a multimedia instructional environment. The learners rated pictures, videos, and animations 
nearly the same. Learners also rated e-learning as an instructor-led learning environment rather highly. Although one 
must be careful not to draw too many inferences from individual items, the learners did report liking some elements 
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(such as the use of animations and hearing the voice of the instructor) that instructors had indicated they did like 
using. 

Learners were more cautious in characterizing the ability of e-learning to be effective learning 
environments or support self-paced learning. The average ratings of these items were close to neutral with some 
even being slightly negative. Learner’s attitudes toward self-regulated learning are an important factor that affects 
performance in an e-learning environment (Chen, 2009). 

Although there was significant correlation between learners’ technology experience and some e-learning 
attitudes (as self-based learning environment and e-learning as instructor-led learning environment), the actual 
relationships were very weak. In contrast, the instructors had much stronger relationships. This may be largely due 
to the asymmetrical nature of e-learning. Creating more advanced and sophisticated content by instructors still 
requires increasing levels of technical competence. However, the presentation of e-learning to students is focused on 
providing access without the added obstacle of additional technical skills. To some extent, this may always be true. 
However, it may also encourage a model where learners are passive receivers of information rather than active 
participants. 

Limitations 

This study was conducted with a convenience sample of instructors and learners from generally one college 
at a single institution. This limits the direct generalizability of the results to other settings. Originally, the researchers 
had also planned to aggregate results by the level of e-learning experience of both the student and instructor 
participants. In both cases, the number of participants and unequal distribution of experience made this difficult. A 
more sophisticated sampling method and larger population would be needed to make this feasible. 

Self-report was used in this study to obtain measures of technology experience and attitudes for both the 
instructors and students. Although there was no reason to believe that the participants weren’t being sincere in their 
responses, self-report can be affected by a number of factors. Also, asking for technology experience and attitudes 
toward e-learning in the same instrument could certainly cue participants as to the intentions of the research. 
Attitudes in particular can be complex and the instruments used were relatively short. Other methods and more in- 
depth research could expand on the relationships suggested here. 

Implications 

The results of this study do help those involved in the planning and delivery of e-learning environments. 
Although certain media do involve more time and effort, learners are certainly drawn to their use. The relatively 
neutral report of effectiveness from learners could come from a number of factors, but it is of great interest. It may 
mean that past experiences with e-learning have been mixed in terms of effectiveness. It may point to deeper issues 
with regard to campuse policies that have generally accepted e-learning as a necessary component in most academic 
areas. There is a need for future research to examine best practices and better pedagogical designs for e-learning 
settings. This study does not provide a definitive judgment on e-learning, nor does it fully explore the complex issue 
of instmctor and learner attitudes. It does however provide additional background for further research. 
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Abstract 

For academics who administer, design, and develop courses for online graduate 
programs, emphasis is placed on creating student-centered experiences that are grounded 
in scholarly learning and instructional theory using state of the art technology. This study 
sought to identify discussion strategies that would enhance the student learning 
experience when integrating a synchronous component into an asynchronous online 
course for graduate students. 

Key words: Discussion; Brookfield; Communities of Inquiry; Blended instruction; Synchronous instruction 

Introduction 

A major paradigm shift is occurring in education as a result of the Read/Write Web. Web 2.0 provides the 
easy and fast dissemination and transference of information allowing for individualized and collaborative learning 
anytime, anywhere and any place as long as there is connectivity. (Bonk, 2009; Christensen, Horn & Johnson, 2008 
Richardson, 2009). Bonk (2009) summarizes the phenomenon in a single sentence: “anyone can now learn anything 
from anyone at anytime” (p.7). Richardson (2009) advises that it is time to reevaluate our traditional teaching and 
learning strategies surrounding how to best teach students. One strategy he suggests is to move away from lecture 
and toward conversation as a way of teaching. 

Brookfield (1995) has been advocating discussion as strategy for the effective teaching of adults. Standing 
on the shoulders of educators like John Dewey, Paulo Freire, and Jurgen Habermas, Brookfield emphasizes the 
importance of ensuring that all interactions are critically reflective whereby students and facidty both are willing to 
come to the discussion with an open mind willing to question long held core values, beliefs and assumptions with 
the goal to create a democratic classroom open to all points of view. Brookfield has been successfully teaching the 
course Discussion as a way of Teaching at Teachers College at Columbia University for well over a decade. 
Brookfield (2007) begins the course with explanations as to why discussions fail including unrealistic expectations, 
lack of student preparedness, lack of ground rules, inappropriate reward systems, and ineffective or no teacher 
modeling. Through creative discussion formats, Brookfield creates a climate for democratic discussion in his 
workshops. Using Richey and Klein’s (2007) definition of design and development research, which includes the 
study of instructional interventions and strategies, this study sought to identify discussion strategies that would 
enhance the student learning experience when integrating a synchronous component into an asynchronous online 
course for graduate students. 

Research Problem Statement 

While webcasts have been used in corporations primarily as broadcast lecture for well over two decades, 
there has been little research on how to successfully integrate synchronous webcasts into online graduate degree 
programs. Most of the research to date surrounds the support needed for design and implementation (Armstrong, 
Bloom, Morris, & Solomita, 2007; Armstrong, Barronco-Morris, & Solomita, 2008; Groen, Tworek, & Soos- 
Gonczol, 2008) or comparing online text-based lectures to webcast lectures (Skylar, 2009). While Hamely (2010) 
also provides guidance on technology support and setting expectations about the webcast environment, he provides a 
few strategic suggestions including re-convergence and student-centered designs that will constantly keep the 
learners engaged through open-ended questions about topics which the learners find relevant. 
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According to Garrison, Anderson, and Archer (2000), oral communications, the cornerstone of traditional 
classroom education is “fast paced, spontaneous, fleeting, and less structured than text-based communications” (p. 
90). While, extensive research has been done by Garrison, et al. (2000) and Garrison, Cleveland-Innes, and Fung 
(2010) on the elements of a Community of Inquiry (Col) that are crucial for student success in higher education in a 
text-based, computer conferencing delivery system, there is little research on how these elements might work in a 
synchronous online environment using democratic discussion as an instructional strategy. By blending online 
synchronous discussion with online asynchronous discussion, an instructional strategy may be implemented that 
leverages the structure, time for deep reflection, and critical thinking associated with asynchronous online learning, 
and the social and emotional characteristics of face-to-face communications in fast-paced, oral discourse. 

Adobe Connect has been used by this online university for several years, typically for faculty and 
administrative meetings. Recently all faculties were provided with Adobe Connect dedicated classrooms and Ready 
Connect conference line numbers. Faculty members are encouraged to create archived webcasts using Adobe 
Connect to welcome students at the beginning of each term, and for live, real-time office hours. Initial response from 
students was positive. The specific problem is that knowledge surrounding instructional and learning strategies to 
successfully implement and integrate synchronous sessions into an asynchronous graduate course is not readily 
available. 

Conceptual Framework 

The concept of community as being integral to learning has been espoused throughout the literature. Lave 
and Wenger (1991) talk about Communities of Practice while others discuss communities of learning and 
Communities of Inquiry (Col) (Garrison, et al.; Garrison, et al., 2010). Three components are at the core of the Col 
Framework: teaching presence, cognitive presence, and social presence. Teaching presence includes those 
responsibilities that traditionally belong to the teacher including: presenting content, creating learning activities, 
assessing learning, creating timelines, facilitating purposeful collaboration and critical reflection, and scaffolding 
learners. Cognitive presence encompasses those activities that are traditionally associated with learning such as the 
process of inquiry and learning through problem solving, looking for relevant information, critical reflection, 
creating meaning, and uncovering solutions. The final element of the Col framework is social presence, which 
embodies the essence of the term community whereby people get together in a trusting, safe environment where they 
can develop personal relationships with peers and teachers (Garrison, et al., 2010). Social presence is demonstrated 
by free and open communications within a cohesive group (Garrison et al., 2000). Although the Col model 
emphasizes the importance of reflection and discourse in the environment, and their 2000 model (see Figure 1) 
includes an element called “Supporting Discourse” that intersects cognitive presence and social presence, there is no 
discussion of the instructional strategies needed to create a specific environment that encourages and supports the 
use of discourse and discussion (Garrison et al., 2000). 
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Conceptual Framework for Integrating Discussion Strategies in a Community of Inquiry 



Community of Inquiry 




Brookfield and 
Preskill’s Nine 
Dispositions of 
Democratic Discussion 


1. 


Hospitality 


2. 


Participation 


3. 


Mindfulness 


4. 


Humility 


5. 


Mutuality 


6. 


Deliberation 


7. 


Appreciation 


8. 


Hope 


9. 


Autonomy 





Figure 1. Adapted from “Critical inquiry in a text-based environment: Computer conferencing in higher education.” 
By D.R. Garrison, T. Anderson, and W. Archer, 2000, The Internet and Higher Education 2(2-3), pp. 87-105. 
Copyright 2000 by Elsevier Science Inc. 



Brookfield and Preskill (1999) define the term discussion broadly as both the theoretical as well as practical 
exploration of group talk covering definitions of dialogue, discourse, and conversation as well as discussion. This 
definition involves two or more individuals participating in either a serious or playful event whereby they are 
actively engaged in mutually agreed upon critique. Purposes for discussion include: (1) to develop a deeper 
understanding of the content domain, (2) to improve self-awareness and ability to critique self, (3) to appreciate 
diverse and culturally different viewpoints that emerge from the group, and (4) to trigger informed action and 
change. Central to creating the right environment in a democratic discussion are the following elements: (1) 
hospitality where everyone feels invited to join in and participate, (2) participation by everyone in as many ways as 
possible, (3) mindfulness where all participants are actively and empathetically listening to one another, (4) humility 
of all recognizing no one has complete knowledge or experience on any topic, (5) mutuality whereby each 
participant encourages every other participant’s self-development as much as their own, (6) deliberation in a 
scholarly manner supported by evidence, data, and logic, (7) appreciation expressed openly by participants to one 
another, (8) hope as expressed by each individual’s desire to learn, gain new perspectives, and clarify conflict, and 
(9) autonomy whereby each individual has the rights to his/her own perspective (Brookfield & Preskill, 1999). 



Research Question and Purpose Statement 

The purpose of this qualitative case study was to describe how synchronous discussion could be effectively 
integrated into an online graduate course that was originally delivered asynchronously. With the Col model and 
Brookfield and Preskills ’s principles of democratic discussion as a foundation, the central research question guiding 
the study was: How can synchronous discussion be integrated in an online graduate course that is primarily 
asynchronous? 

Methodology with Limitations 

In one winter 2011 education course, students were invited voluntarily to attend a two-hour Adobe Connect 
Session on a Saturday afternoon. It was decided that the format for the session would be discussion based on 
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Brookfield’s (2004) Circular Response Discussion and Circle of Voices discussion techniques. A formative 
evaluation using qualitative responses from 14 student participants in the initial session pilot were collected using a 
Critical Incident Questionnaire (CIQ) (Brookfield, 2004). The CIQ is in Appendix A. Feedback from the pilot was 
used to design and develop a more in-depth synchronous component for spring 2011. 

Richey and Klein (2007) emphasize that their definition of design and development research includes the 
study of instructional interventions during program development as part of a systematic approach to establishing an 
empirical basis for the creation of instructional products, tools, and interventions. Design and development research 
is always applied research seeking to understand what actually occurs in practice. For program development design 
research, they recommend (1) field testing the program with evaluation procedures, (2) providing evidence of 
successful learner outcomes as well as learner motivation, (3) collecting usability data, (4) relating program design 
to evaluation results, (5) relating program design to learning outcomes, and (6) relating program design to learner 
motivation. Following the pilot study and Richey and Klein’s advice for structuring program development research, 
a qualitative case study of 13 higher education graduate participants was conducted by administering a CIQ 
questionnaire following each of the 6 synchronous sessions held during the 10-week course in a course on delivery 
of distance education. Data analysis included in-depth review of students’ responses on these questionnaires as well 
as student posts in the final week of the course evaluating the course, and anonymous student responses on 
university implemented end-of-course evaluations. 

During the 10-week graduate course, the six Adobe Connect sessions were integrated into an online 
asynchronous graduate course on delivery of distance education. Only one of the sessions, week #3, was listed in the 
syllabus as required. The others were optional, but could be substituted for traditional response posts to weekly 
asynchronous discussion sessions. 

Adobe Connect Sessions were held in weeks 2, 3, 4, 5, 7, and 9. Using two of Brookfield and Preskill’s 
democratic discussion strategies, Circle of Voices and Circular Response discussions, students were required to 
participate following specific rules and guidelines adapted from Brookfield’s (2007) course. The general ground 
rules for all discussion held in the synchronous session included: 

• Speak one at a time for two minutes as called by the facilitator. 

• No interruptions. 

• No criticism of anyone’s answers, even if they are incorrect. 

• Mutual respect by and for everyone in the discussion. 

• Communicate in the clearest way possible. 

• Provide the most accurate/comprehensive answer as possible. 

• Genuinely try to be understood by others. 

The specific ground rules for Circle of Voices included: 

• Speak one at a time for two minutes as called by the facilitator. 

• Begin in order of Adobe Connect participant list. 

• Every one contributes. 

• Two minutes of silent time to organize thoughts. 

• Discussion opens. 

• Each person contributes with uninterrupted air time. 

• No interruptions while each person speaks. 

The specific ground rules for Circular Response included: 

• Speak one at a time for two minutes as called by the facilitator. 

• Begin in reverse order of Adobe Connect participant list 

• Every one contributes, but must incorporate: 

• At least one reference to preceding speaker ’s message 

• Agreement or dissent with the previous opinion 

• If no point of connection, reason for the source of confusion such as gap in knowledge, experience 
or language used 

• Two minutes of silent time to organize initial thoughts. 

• Discussion opens, one minute for each speaker. 

• Each person contributes with uninterrupted air time. 

• No interruptions while each person speaks. 

Following each synchronous Adobe Connect Sessions, students posted CIQs (Appendix A) with their 
perceptions of the experience. Student responses were coded and changes were made to the format of the 
discussions for the next sessions based on their feedback. 

As Richey and Klein (2007) observed, in most traditional research methodologies, the roles of the 
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researcher and the participant are separate and distinct. However, in design and development research, this is often 
not the case and the researcher is often the designer or developer. That is what happened in this study. Richey and 
Klein caution researchers in this dual role and advise the use of strategies to minimize errors in results such as 
setting up procedures to segregate the data and triangulating data from multiple sources to prevent researcher bias. 
Both of these strategies were implemented. Strict procedures were set up to collect student CIQs and code them 
using the pseudonyms Student 1, Student 2, etc. Questionnaires were separated by weekly unit. Triangulation was 
achieved through the literature review, the CIQ questionnaires, the pilot study, and analysis of the end of course 
student evaluation threads as well as the end of course online student evaluations. 

A preliminary list of start codes (Miles & Huberman, 1994) was created from the conceptual framework, 
the literature, and the researcher’s experience. As a foundation for the Col components of the framework, the 
Community of Inquiry Coding Template developed by Garrison et al. (2000) was used as a tool to analyze the CIQs 
for the presence of cognitive, social, and teaching presence in the synchronous online sessions. Brookfield and 
Preskill’s nine Dispositions of Democratic Discussion were used as the basis for analyzing the CIQs for the presence 
of democratic discussion. Data was analyzed using NVivo 9 to search for themes from the literature and emerging 
themes. Findings and results were presented at the November 2011 conference of the Association for Educational 
Communications and Technology (AECT). 

While the study allowed for the exploration and identification of discussion strategies that can be used to 
enhance the student learning experience and motivation when integrating a synchronous component into an 
asynchronous online course for graduate students, the study has several limitations. 

• The researcher’s role as facilitator in the course may have prevented some of the student 
participants from being as honest as they might otherwise have been in filling out their CIQs. 

• Case studies by their very nature present a number of limitations including sample size and 
timeframe. This study involved 13 learners in one section of a graduate course lasting ten weeks. 
Both of these factors prohibit the generalization of the findings to the overall population. 

Results and Findings 

With the exception of week 3, which was labeled mandatory in the syllabus, all Adobe Connect Sessions 
were optional. Table 1 shows student participation by week for the percentage of times each student participated, 
participation totals for each session, and the percentage of students attending each session. 

Table 1. Participation in Adobe Connect Sessions by Week 



Participant 


Week 2 


Week 3 


Week 4 


Week 5 


Week 7 


Week 9 


Total 


Percentage 


Student 1 


X 




X 


X 


X 


X 


5 


83.3% 


Student 2 


X 




X 


X 


X 


X 


5 


83.3% 


Student 3 










X 




1 


16.7% 


Student 4 


X 


X 


X 


X 


X 


X 


6 


100% 


Student 5 


X 




X 


X 


X 


X 


5 


83.3% 


Student 6 


X 


X 


X 








3 


50% 


Student 7 


X 


X 










2 


33.3% 


Student 8 


X 






X 






2 


33.3% 


Student 9 


X 












1 


16.7% 


Student 10 


X 












1 


16.7% 


Student 1 1 


X 


X 




X 


X 


X 


5 


83.3% 


Student 12 






X 


X 


X 




3 


50% 


Student 13 


X 


X 


X 








3 


50% 


Totals 

Percentage 


11 

84.6% 


5 

38.5% 


7 

53.8% 


7 

53.8% 


7 

53.8% 


5 

38.5% 


42 





Of the thirteen students in the course, 1 student attended 100% (6 out of 6) of the sessions; 4 students attended 
83.3% (5 out of 6); 3 students attended 50% of the sessions (3 out of 6); 3 of the students attended 16.7% of the 
sessions (1 out of 6). Two of the 3 students who attended only once also received Incompletes for the course. 

Student responses on the CIQs were evaluated to understand the importance placed by the students on democratic 
instruction, social presence, cognitive presence, and teaching presence. Results are shown in Table 2. All students 
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(13 out of 13) cited democratic instruction, social presence, and cognitive presence as important to their educational 
experience during the Adobe Connect sessions. Teaching presence was cited by 76.9% (10 out of 13) of the students 
as being important to their educational experience during the Adobe Connect session. 



Table 2. Elements Rated Important in Adobe Connect Sessions 



Participant 


Democratic 

Instruction 


Social 

Presence 


Cognitive 

Presence 


Teaching 

Presence 


Total 


Student 1 


X 


X 


X 


X 


4 


Student 2 


X 


X 


X 


X 


4 


Student 3 


X 


X 


X 




3 


Student 4 


X 


X 


X 


X 


4 


Student 5 


X 


X 


X 


X 


4 


Student 6 


X 


X 


X 




3 


Student 7 


X 


X 


X 


X 


4 


Student 8 


X 


X 


X 


X 


4 


Student 9 


X 


X 


X 


X 


4 


Student 10 


X 


X 


X 


X 


4 


Student 1 1 


X 


X 


X 


X 


4 


Student 12 


X 


X 


X 


X 


4 


Student 13 


X 


X 


X 




3 


Totals 

Percentage 


13 

100% 


13 

100% 


13 

100% 


10 

76.9% 


5 



Two scenarios caused learners frustration during the Adobe Connect sessions. The first was situations when 
the technology infrastructure did not work correctly. This happened one week when breakout rooms were tested and 
did not work well with the telephone conference service. The second was when students did not have adequate 
technology to effectively participate in the sessions. Often this took added time away from the discussions. This 
issue diminished in later sessions. 

During the 10-week term, a Col was created whereby students envisioned they were active participants in a 
cohesive community where they could readily share emotions, knowledge, experience, and ideas with their peers 
and their teacher. To foster community, students perceived that it was critical that the faculty member be 
“welcoming,” and accepting of the students’ facility with technology and the course content. They expressed 
enhanced learning when faculty validated their own understanding and when faculty added new perspectives to the 
discussion. 

Certain aspects of democratic instruction were highly important to the students. These included hospitality, 
the fact that everyone was welcome and invited to join; participation, the fact that everyone who attended 
participated in every discussion; mindfulness, all participants actively and empathetically listening as others spoke; 
and deliberation, whereby everyone came prepared for the discussion and supported their position with evidence 
from the research literature. To increase the effectiveness of the weekly discussions, the students found that the 
format and organization of the weekly discussions to be critical. They preferred to know the topics in advance and 
have a detailed agenda, rather than ad hoc discussion. They liked the fact that everyone participated equally. They 
liked having ground rules and added to those throughout the early weeks. They found that they learned significantly 
more when they were well prepared and when the other students were well prepared. Students became anxious when 
other students came in late, were unprepared, talked off topic, or participated in discussions using acronyms that 
were unfamiliar. 

Conclusions and Recommendations 

Using Adobe Connect, a synchronous virtual classroom technology, to augment an asynchronous online 
graduate course using Brookfield and Preskill’s instructional strategies for democratic instruction can be very 
effective for increasing involvement and motivation of graduate students in higher education. Structuring the 
program so that the synchronous session’s leveraged student required research and study activities for the regular 
weekly assignments had two benefits. First, students did not have additional preparation time for the synchronous 
sessions, and, second, students had a second opportunity to validate their understanding of the weekly objectives 
through critical reflection and empathetic listening. Based on this experience, two recommendations are clear: (1) 
care must be taken to ensure that the sessions are well planned, well organized, and effectively facilitated, And (2) 
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the principles of democratic instruction must be followed. Further research might include using more of Brookfield 
and Preskill’s discussion strategies and other discussion-based instructional strategies such as Action Learning. 



Significance for Practice 

Leaders in higher education are looking for ways to improve student retention, persistence, and success. 
Blending online synchronous instruction into online asynchronous graduate courses may provide students with the 
connection to each other, to the teacher, and to the institution which is so difficult to achieve in strictly asynchronous 
programs. 
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Appendix A 



The Classroom Critical Incident Questionnaire 

Please take about five minutes to respond to each of the questions below about this week's Adobe Connect 
synchronous class (es). If you want your response to be anonymous to the rest of the class, send it to me at my email 
address; otherwise, post in the discussion thread. Thanks for taking the time to do this. What you write will help me 
make the class more responsive to your concerns. 

At what moment in class this week did you feel most engaged with what was happening? 



At what moment in class this week did you feel most distanced from what was happening? 



What action by anyone (teacher or student) in class this week did you find most affirming or helpful? 



What action by anyone (teacher or student) in class this week did you find most puzzling or confusing? 



What surprised you the most about the class this week? (This could be something about your own reactions to what 
went on, or something that someone did, or anything else that occurs to you). 



Used with the permission of Stephen Brookfield. Adapted from his work for use in an Adobe Connect Synchronous 
Session. Retrieved from 

http://www.stephenbrookfield.com/Dr._Stephen_D._Brookfield/Critical_Incident_Questionnaire_files/CIQ.pdf 
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Abstract 

An intervention module was used to provide the basic concepts related to critical thinking, creative thinking, and 
collaborative thinking processes for pre-service teachers. Following the delivery of 21 st century skills concepts along 
with an overview of the classic TPACK model, students designed lessons that incorporate both pedagogical and 
content area decisions. An interweaving theme emerged showing an interesting interplay between basic pedagogy 
and 21 st century skills framework. This interweaving of critical, creative, and collaborative thinking could make the 
difference in preparing pre-service teachers to be more effective in design of classroom instruction. 

Introduction 

The TPACK model provides a workable solution for technology integration in the K12 classroom. There 
are many reports to support the validity of TPACK for designing lessons for good fit within specific K12 content. 
But, we want more than efficient delivery of instruction in content areas. The 21 st century demands creative, critical 
thinking that leads to problem solving within a community. The 21 st Century Skills framework recommends higher 
order thinking through the “4 C’s” including critical and creative thinking; collaborative processes and problem 
solving. Likewise, the new Common Core Standards focus on complex thinking that will prepare students for 
successful entry into college classrooms and careers. An analysis of pre-service teachers’ lesson plans may reveal 
the potential for TPACK as a foundational framework for using higher order thinking skills recommended for 21 st 
century learners. 

Elements of TPACK. 

According to Mishra and Koehler (2006) effective use of technology in classroom requires a complex 
interplay across several instructional elements. These include understanding in content and pedagogy; technology 
and pedagogy; and content supported by technology. Decisions are based on the attributes of a particular content 
area. For today’s classrooms, technology provides many affordances to ensure effective knowledge representation, 
hence the interplay between content, pedagogy, and technology knowledge (Mishra & Koehler, 2006; Koehler, 
Mishra, & Yahya, 2007). To achieve expected outcomes, the interplay must be systematic in design. As in any 
workable system, each part depends on all the others and each contributes to the good of all. The proposed 
framework that brings together content, pedagogy, and technology into workable solutions for classroom instruction 
is called TPACK or Technology, Pedagogy, and Content Knowledge. Teacher educators are aware of the need to 
prepare new teachers to be successful in the use of technology for 21 st century classrooms. This goal is reached 
through “powerful ideas about teaching and learning” (Mishra, Koehler, & Kereluik, 2009 p. 49), rather than the 
continual training in the latest new thing. Following in this line of research, we identified a need to investigate how 
well pre-service teachers can use the TPACK model to design instruction while also incorporate skills and concepts 
needed for contemporary learning environments. Our interests were focused on themes presented in the current 
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standards for Pre-K12 classrooms and how these might add to the complexity of TPACK to prepare future teachers 
entering the 21 st century classroom. 

Complexity, Common Core and 21 st century 4 C’s. 

Recently published Common Core Standards (CCS) are in the developmental stage and include 
competencies in two content areas— Mathematics, and English Language Arts (OSPI, online July 2011). The 
common core emphasizes conceptual understanding by bringing together a balance of pedagogy and core content for 
K12 classrooms (Magner, 2011). The standards were developed based on the work by the National Governors 
Association Center for Best Practices (NGA Center) and the Council of Chief State School Officers (CCSSO). The 
goal to provide a common curriculum for all 50 states will soon be achieved. At this writing, 44 states have adopted 
the CCS and many have begun implementation through professional development for K12 teachers. In comparison, 
the Partnership for 21st Century Skills is a consortium whose goal is to educate teachers and the community in 
attributes for 21st century tools, methods, and necessary learning environments. As we began this study, we saw 
value in each set of standards. Advocates for Partnership for 21st Century Skills (http://www.p21.org/) emphasize 
the effective use of tools and resources to fuse core subjects with themes related to global learning, civic, health, and 
environmental literacy along with critical thinking, problem solving, communication and creativity. There are 
reports that P2 1 .org is under the control of private organizations in an effort to ensure ongoing spending for 
technology in the schools (Sawchuk, 2009). Not to be outdone, opponents of the Common Core suggest too much 
national power overrides local decision makers for school curriculums (Sloan, 2010). 

Rather than focus on controversy and comparison of the standards, our purpose is to establish a theoretical 
foundation for thinking skills best applied to K12 curriculum. In fact, there are many states working to implement 
Common Core Standards in concert with the themes for 21st Century Skills (Education Week, 2011). Elements 
within the Common Core clearly represent processes for understanding in context with increasing complexity, 
reflection, and critical thinking. For 21 st century students, the tools for complex thinking processes require unique 
skills, both tactile and dispositional that meets the needs of a global-information-driven society. Corresponding 
elements for critical, creative, and complex thinking skills appear in both sets of standards. 

The four C’s: Critical thinking, creativity, and collaborative communication. 

Fawkes, O’Meara, Weber, and Flage (2005) examined the design of The California Critical Thinking Skills 
Test (CCTST). As part of their research, a summary was prepared listing main indicators deemed important for 
secondary level students These included skills and ability to interpret and apply complex texts, follow instructions, 
make distinctions across different ideas and concepts, draw conclusions, write or interpret premises, make 
assumptions, assess the relevance of claims presented by others, and evaluate deductive arguments. The CCTST 
website (http://www.insightassessment.com/About-Us) lists additional indicators of critical thinking skills. These 
include inquisitiveness with regard to a wide range of issues, motivation to remain well-informed, alertness to 
opportunities for critical thinking, self confidence, flexibility, fair-mindedness, honesty in facing bias, and 
willingness to reconsider following honest reflection. The CCTST group also suggests more emphases be placed on 
collaborative skills and dispositions. 

The Partnership for 21 st Century Skills (http://www.p21.org), a consortium consisting of educators and 
entrepreneurs, also recommend classroom instruction based on the attributes for critical, creative, collaborative 
thinking. Classroom teaching practices aligned with the P2 1 .org skills framework will focus on analysis and 
evaluation of information, synthesis leading to connections across ideas, ability to interpret information and draw 
conclusions. Students should also be able to reflect on their own learning experiences. Many of these characteristics 
are applied to creative and collaborative thinking processes. The ability to create new and worthwhile ideas, 
elaborate, refine, analyze and evaluate one’s own ideas while working creatively and effectively with others 
(collaboration) are fundamental to the skills recommended by P21.org. 

Being able to learn, create, solve, and generate ideas within a collaborative learning environment reflect the 
job skills inventory being published by prospective employers of today (Cox, Aim, & Holmes, 2004). In addition, 
legislators are listening to educators who see the value in critical thinking in today’s learners. Senator Kay Hagan 
(D. NC) has introduced legislation “that supports and encourages the hard work of P21's sixteen 21st Century 
Leadership states (Walker in Hagan, online).” Hagan and others wish to provide students with the knowledge and 
skills to be prepared for college, careers, and citizenship in today's global economy. Thus it seems reasonable to 
identify connections among 21 st century skills and elements within the technology, content, and pedagogical 
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decisions proposed by Mishra and Koehler (2006). Clearly, there is a valid need for teachers to plan use of 
technology based on new century skills integrated within core subjects. We ask, “How well can pre-service teachers 
adopt the practices needed to make wise pedagogical decisions based on the type of content along with use of 
technology to enhance the learning experience? Are pre-service teachers able to design complex instructional 
activities that also apply the higher level thinking processes represented by the four C’s? 

Need for the Study. 

There are two questions to be answered through this study. First, we asked if use of the well known 
TPACK model might provide the framework needed to infuse higher order thinking within pedagogical decisions 
for lesson planning, and second would the design of lessons, by undergraduate pre-service teachers, reflect 
systematic processes resulting in good fit across content, pedagogy, and technology? We began by defining themes 
within the 21st century skills framework for critical thinking, creative thinking, and collaborative processes 
(www.p21.org). Based on these findings, we could make recommendations for preparing teachers to focus on 
content and pedagogy while selecting technology to transform content so it is accessible to K-12 learners. More 
importantly, content can be learned using critical, creative, and collaborative thinking processes. We propose 
TPACK model interwoven with 21st century thinking skills could make the difference in preparing pre-service 
teachers to be more effective in design of classroom instruction. 



Methods. 



Participants 

The study was conducted in a large regional university with nationally recognized programs in elementary 
and middle grades education. These programs require a course designed to prepare teachers in the effective use of 
technology for their future classrooms. All participants were pre-service teachers (n=139) in a variety of teacher 
education programs including elementary education, middle grades, and some subjects in secondary education. 
Female participants (n= 113) outnumbered males (n=26). Students were enrolled in a technology integration course 
required for their program of studies. 

Intervention 

The researchers, one of whom was course instructor, developed modules on TPACK model and basic 
attributes for the four C’s described above. To ensure higher order thinking as part of the design of their lessons, 
participants were instructed to include at least one of the following — critical thinking, creative-innovative thinking, 
collaborative-communicative thinking, and problem-solving within their instructional activities. The lesson were 
submitted to the course instructor and reviewed by four evaluators. These consisted of researcher 1 who developed 
the modules for designing TPACK lessons; researcher 2 who designed the lesson template and guided participants 
through the design process; a master teacher with over ten years experience in teaching high school mathematics and 
working as a building level technology facilitator; and a master teacher in a second grade classroom. Instructions for 
use of the rubric and ten practice sets were completed by the master teachers prior to actual analysis of the lesson 
plans. 

Using a collapsed version of TPACK rubric validated by Harris, Grandgenett, and Hofer (2010), two 
researchers and two master teachers assessed lessons (N =113) for good fit across content, pedagogy, and 
technology. Qualitative analysis of each lesson was conducted to identify evidences of the 1 ) lesson activities 
aligned with the curriculum (Content); 2) activities leading to creative/critical/collaborative thinking processes to 
master content (Pedagogy); 3) effectiveness of tools to support content and teaching methods (Technology); and 4) 
overall good “fit’’ for the three components. A total 113 lessons were usable for the qualitative analysis. Scores on 
the rubric ranged from 1 to 4 with 4 being the highest. An inter-rater reliability analysis using the Cohen Kappa 
statistic was performed to determine consistency between raters’ overall mean score for each lesson plan (Kappa = 
0.603). As a rule of thumb values of Kappa from 0.60 to 0.79 is considered substantial (Landis & Koch, 1977). 

Each lesson was read and analyzed separately and evidences related to the four themes were then coded as 
an independent activity by each researcher (Patton, 1980). The two researchers combined two separate levels of 
expertise during the analysis, one being a purist in area of instructional design and the other having extensive 
experiences in best practices for integration of technology in K12 settings. Combined analyses of all lessons show 
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tallies for the themes. The master teachers analyzed lessons based on instructional modules provided during training 
in use of the rubric. Critical thinking was identified using attributes described in the 21st Century Themes as 
described in the www.p21.org website and attributes described in the professional literature in critical thinking skills 
(Anderson & Krathwohl, 2001; Facione & Facione, 1994; McMahon, 2009). Critical thinking was defined as: 
Reason effectively, use systems thinking, and make judgments and decisions (P21.org, online). Similar criteria were 
used to identify creative thinking with attributes described in the literature on creative-innovative thinking activities 
(University of Georgia-Creativity Resources, online). Creative thinking is defined as the ability to use a wide range 
of idea creation techniques; create new and worthwhile ideas; evaluate, refine, and analyze one’s own ideas 
(P21.org, online). Communicative thinking was defined as: articulate thoughts and ideas effectively using a variety 
of communication modes. Collaborative communication requires the design of group processes with strategies for 
reporting outcomes from the work completed by the group (Johnson, Levine, & Smythe, 2009). 

Analyses and Findings. 

See in Table 1 rubric headings used to analyze and evaluate lessons plans submitted by the pre-service 

teachers. 



Curriculum goals & 
Technology 
(Curriculum-based 
technology use). 
*Criterion One* score. 


Instructional Strategies & 
Technology (using 
technology in teaching 
learning). *Criterion Two* 


Pedagogy & content 
(Pedagogical decisions 
support critical, creative, 
collaborative, and/or 
problem-solving thinking 
(21stC). *Criterion Three* 


"Fit" (Content 
pedagogy and 
technology 
together). 
*Criterion Four* 



Table 1. Rubric used to evaluate use of technology, instructional strategies, the 4 C’s and goodness of fit. Scale is 1 
to 4 with l=lowest and 4 = highest 

Criterion One refers to appropriate use of technology based on content. Criterion Two is best use of 
technology for a particular instructional strategy. Criterion Three refers to pedagogy & content, however the 
original rubric (Flarris, et al, 2010) was modified to take into account 21 st century skills. Evaluators rated 
pedagogical decisions to support critical/evaluative thinking and added to evaluate the use of 21 st themes for critical, 
creative, collaborative communication as part of each lesson (P21.org). All four evaluators were familiarized with 
attributes of the 4 C’s and used reviews from the literature to add to the knowledge base on these themes as 
described in www.p21.org. Criterion Four is used to evaluate the over-all good fit across content, pedagogy, and 
technology evident within the lesson plan. 



One: Curr Goals 


Two: Use of Tech 


Three: Pedagogy 


Four: Good fit 


Grand M 


3.515 


2.855 


3.095 


2.967 


3.15 



Table 2. Mean scores based on researchers combined evaluation of TPACK framework and 4 C’s in 21st century 
classrooms. 

As can be seen in Table 2 lesson plans with curriculum goals matched to use of technology received the highest 
ratings (m = 3.515). Evaluation of criterion three was based on the creative, critical, and collaborative- 
communication thinking skills used in the lesson. Using the definitions established prior to qualitative reviews, the 
evaluations of pedagogy resulted in a mean score of 3.095. Evaluators rated use of technology and overall “good fit” 
with lower scores. Use of technology was evaluated on instructional activities to promote thinking processes and/or 
appropriate knowledge representations (m=2.85). Good fit was evaluated on design of lessons that make good 
choices based on attributes of content area, followed by pedagogical decisions (m=2.967). 
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Further analyses matched the themes for the 4 C’s to instructional strategies. Table 3. provides a tally of 
instructional strategies that were identified within the 113 lessons analyzed. 



Fligher Level Bloom’s 
Creative, critical, or 
collaborative thinking 
skills 


Frequency 




Lower Level Bloom’s' 
Taxonomy 
Comprehension, 
Application. 


Frequency 


T eacher-Questioning 


41 




Modeling by teacher 


29 


Compare and contrast 


29 




Group discussions 


26 


Inquiry based 


18 




Journaling 


26 


Problem or project based 


16 




Internet searches 


13 


Student presentation 


12 




Teacher demonstration 


12 


Analyze information 


7 




Drill and practice- 
paper and electronic 


12 


Concept mapping 


7 




Guided reading 


8 


Brainstorming; student 
generated lists 


9 




Lecture 


8 


Scenario; real-world 
application; role playing 


5 




Note-taking (with and 
without guides) 


8 


Peer teaching 


3 




Examples and 
nonexamples 


3 


Perspective taking 


2 




Guided practice 


3 


Graphing 


1 




Independent practice 


1 



Table 3. Frequency of instructional strategies coded by level of higher order and lower order thinking skills. 

A qualitative analysis of strategies within lesson plans show frequency of critical, creative, and collaborative 
thinking processes compared with lower level thinking defined by Bloom’ s taxonomy (Anderson & Krathwohl, 
2001). The predominant instructional strategy was use of “teacher-questioning”. In some cases, the lesson plan 
included sample questions to be used in the classroom, thus the researchers could determine the level of questioning 
and suitability for curricular goals. Clearly, expertise in use of open-ended questioning leading to critical analyses 
would be desirable in this type of lesson activity. Following “teacher-questioning”, inquiry based learning strategies 
were used. Use of open-ended questions to research a particular topic appears frequently in pre-service teachers’ 
lesson planning. Problem or project based learning was also commonly used. These lessons included an authentic 
problem requiring sustained research and reporting back to peers. Use of presentation software (PowerPoint) is a 
favorite with pre-service teachers. There was also frequent use of blogging software and podcasting as a method of 
reporting from project based learning activities. Lessons also included activities requiring brainstorming in group 
discussions. Role playing and peer teaching were also used, but these methods were infrequent. Questioning 
strategies, inquiry based, problem based or project based, analysis of information, and group interaction all require 
use of the 4 C’s as described in the literature review. 

Activities resulting in lower-level thinking processes were frequent. Many of the lessons included modeling 
by the teacher demonstrating cognitive processes or tactile skills. Group discussions were frequent but examples of 
the guiding questions were simplistic and lacked cognitive processes leading to critical analysis, argument, or 
perspective taking. For example, many lessons required students to talk about what was heard in a lecture or 
following assigned reading. Discussions from these activities would lead to recanting of content lacking deep 
analysis. Other activities that include drills, note-taking and guided practice were infrequent. A closer look at the 
higher level thinking activities was charted and is presented in Table 4. 

The 4 C’s: Interpretations by Pre-service Teachers. 

As part of the evaluation of lesson plans, both frequency of higher level thinking strategies as well as 
quality of the design of each lesson were assessed. Table 4 provides a list of strategies identified in pre-service 
teachers’ lesson plans. 
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Thinking skills Frequency 

*Critical 167* 

Critical and collaborative/communicative 54 

Critical and creative 37 

Critical and problem solving 7 

*Creative 108* 

Creative and collaborative/communicative 47 

*Collaborative/Communicative 155* 

Collaborative/Communicative and problem 
Solving 15 

Problem solving 75 

Problem solving and creativity 1 



Table 4. Incidence of thinking skills following qualitative analysis of lesson plans. 

A qualitative analysis of all lesson plans revealed variety of instructional strategies, tools, and levels of thinking for 
the K12 student. Drill and practice activities using an interactive white board was commonly used within the 
lessons. Higher order thinking was fostered through the use of creative writing; reflective thinking and perspective 
taking through word processed essays. Pre-service teachers demonstrated the ability to make appropriate 
pedagogical decisions based on content and selection of technology to enhance thinking or for appropriate 
knowledge representation. This was especially evident in mathematics lesson. Science content often included the use 
Internet websites to gather weather data. There was little mention of lessons using data from other sources or in use 
of probeware for direct data entry. Social studies and history lessons included instructional strategies in perspective 
taking through the use of digital primary sources. Historical events were reviewed and analyzed using streaming 
video. Online cartoon generators provided useful tools for lessons with analyses of history. 

Interpretations. 

An important tenet associated with TPACK is selection of technology tools following the decisions for pedagogy 
and content. As can be seen in Table 2, evaluators rated the over-all good fit across content, pedagogy, and 
technology with lower scores than scores assigned to criterion one — matching technology to curriculum goals or 
criterion two — matching technology to pedagogy. The lower mean score is the result of lessons that selected tools or 
described use of technology in isolation rather than in concert with content and pedagogical decisions. Although 
participants frequently made good decisions for selecting the appropriate hardware and software and provided good 
rationales for the design of their lessons, there was limited discussion and reflection in pedagogical decisions leading 
to student achievement. The lessons lacked references to planning activities and strategies that might result in 
thinking skills deemed important for new century learners. 

As part of the evaluation of lesson plans, both frequency of higher level thinking strategies as well as 
quality of the design of each lesson was assessed. Quality is based on alignment with TPACK framework and the 
attributes of systematically designed instruction. Scores by evaluators indicate pre-service teachers demonstrate 
better knowledge and understanding in selection of best practices to support a particular learning goal than in their 
ability to match learning goal with appropriate technology tool. Ability to systematically plan (good fit) for learning 
goals matched to appropriate methods and selection of technology received the lowest scores by evaluators, thus 
applied use of TPACK seems to need further instruction and practice. Pre-service teachers might benefit from 
cognitive modeling in how to think through the process for selecting instructional methods followed by decision- 
making in selecting a particular tool, and lastly, reflecting on how well content, pedagogy, and technology will 
impact student learning. 

However, the value added to the study is inclusion of both critical- and creative-communicative thinking in 
small groups. Pre-service teachers show higher level of understanding for design of lessons that require 
communication within groups. It was discovered that most commonly used teaching methods included critical 
and/or creative thinking combined with collaborative-communicative thinking processes. Many of the lessons 
included innovations for reporting outcomes from a small group activity. Pre-service teachers are motivated to use 
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presentation software available on Web 2.0 and frequently included the use of such resources as blogs and 
podcasting. Lessons were less likely to include critical analysis of digital primary sources or use of spreadsheets to 
critically analyze numerical data. Many of the lessons were inquiry based which means K12 students would have 
opportunity to explore Web 2.0 resources and creatively report outcomes, however, methods in how a K12 student 
might conduct critical analysis of the information gathered from the resources was less frequent. One notable 
exception was the lesson plan using data collected from weather websites. Analyses of the lesson plans *did* reveal 
attributes of critical, evaluative thinking; creativity and innovative ideas; group processes which lead to open 
communication and successful outcomes for each student. Also, the analysis showed an *overlaying* of the 
attributes such as creative thinking and collaborative thinking. 

Conclusions and Recommendations for Future Study. 

Results of this study suggest limitations in “good fit” across content, pedagogy, and technology in the 
design of lessons by pre-service teachers. Pre-service teachers need sustained modeling in how to select technology 
based on attributes of content paired with pedagogical decisions as first steps in lesson planning. An emerging 
theme suggests pre-service teachers *do* understand how to plan instruction for small groups to think critically 
about a phenomenon, however, pre-service teachers have some limitations when planning systematically. Lesson 
plans lacked flow of thought from learning objectives to pedagogical decisions to methods for assessing outcomes. 
Further research is needed in methods for guiding pre-service teachers’ pedagogical decisions that make the best use 
of technology to achieve higher order thinking. A more in-depth analysis of pre-service teachers’ lesson plans could 
show propensity of a particular content area in use of higher order thinking as defined by the P21.org and the 
Common Core Standards. Future projects might provide in-depth instruction in how to design lessons that include 
individualized critical thinking and creative thinking activities. The design of pre-service methods courses as well as 
technology skills-based courses should be followed by assignments for lesson planning. Evaluation of the lesson 
plans could focus on specific elements using a validated rubric to evaluate thinking skills and goodness of fit across 
content, pedagogy, and technology. Goodness of fit includes appropriate selection of tools to support pedagogical 
decisions already established in the lesson planning process. 

A pattern in use of certain tools and resources emerged during the analysis of lesson plans. Lessons 
included a very high incidence in the number of uses for interactive white boards to display games used as 
simulations, games for drill and practice, and presentation tools such as PowerPoint and Voice Thread. Many of the 
pre-service teachers included online games and simulations to provide elaboration for a particular skill or concept. 
More research in use of interactive games and how these should be selected to enhance critical thinking would add 
to the research in immersive game technology. 

Finally, with increasing interest in digital literacy, the Read-Write Web with literal access to a world of 
information could provide the incentive needed to design lessons with critical analysis of information for solving 
problems familiar to the student. For future research we recommend a focus on communicative-creative activities 
within virtual environments or with the use of interactive Web 2.0 tools. 
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Abstract 

Very often, instructors will make efforts to enhance the learning process of their courses by including 
hands-on classroom activities that support active engagement of their students throughout the course. This paper 
discusses how using the framework proposed the Activity Theory (Engestrom,1987) helped with the effective 
implementation of a set of instructional tools. Wiki and reflective discussion forums to improve the instructional 
process in an elective Chemical Engineering course. 

To measure the impact of these tools we used a mixed-methods approach. A series of t-Test analyses 
indicated an increased perceived engagement and a decreased perceived difficulty as a result of engaging in this 
course. These findings are well complemented by students’ answers on the open-ended questions related to both 
Wiki and discussion forums. The positive findings revealed by students’ perceptions indicate that by integrating in a 
meaningful way technology-driven instructional tools that complement active learning strategies that the overall 
learning experience can be enhanced for most students. 

Motivation of the Study 

Current perspectives on effective engineering and science education emphasize the need for more active, 
hands-on and reflective activities that will help students build skills that will allow them use rather than pilling up 
professional content (e.g. Sheppard et al., 2009). Following these guidelines, instructors often make efforts to 
enhance the learning process of their courses through hands-on classroom activities that support active engagement 
of their students throughout the course. Some examples of such activities include in-class small-group tasks, 
semester-long team projects, guest experts or expert-driven exercises. However, because of the process-oriented 
nature of these tasks, instructors often struggle to find tools that will allow them to: a) monitor and increase the 
effectiveness of these strategies, and b) create the premises for a fair evaluation of each student’s engagement into 
the learning process. This paper discusses how the framework proposed the Activity Theory (Engestrom,1987) 
helped with the effective implementation of a set of instructional tools. Wiki and Reflective Discussion Forums, to 
improve the instructional process in an elective Chemical Engineering course. 

Instructional Context and Strategies 

Hazardous Materials Management, the focus of this study, is an elective course that attracts mostly seniors 
and few juniors. Although most of the students engaged in this course will never be a “first responder” to a 
hazardous materials incident, some of them may become involved in the incident response teams at the 
manufacturing facility that they will work for later on. The major objective of this course is to introduce students to 
the fundamental principles of hazardous materials handling and management and their application in real life 
contexts. To achieve this goal, the instructor implemented an active learning model built on a series of student and 
learning-centered tasks. For the in-class lecture -driven activities, these active learning tasks include: a) use of 
clickers during the lectures; b) small-group in-class discussions of different hazardous materials incident scenarios; 
and c) peer-presented lectures. The major non-lecture active tasks are: a) two hands-on weekend laboratories 
focusing on hazardous material incident response, and b) a semester-long team project. 

The hands-on laboratories provide students with a clear view of what professionals do to manage in an 
effective manner a hazardous materials incident. Two incident response trainers designed and implemented these 
hands-on laboratories. During these laboratories, students assume specific roles as part of a teamwork-based incident 
response scenario. The term project, on the other hand, challenges students to develop a realistic hazardous materials 
release scenario and integrate the knowledge and skills gained during the semester in suggesting an effective 
management strategy for the proposed incident. 
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Technology Tools to Optimize the Effectiveness of Instructional Strategies 

One major focus of this study was to ensure an optimal introduction and use technology tools in the target 
course. The conceptual model proposed by the activity theory provided effective descriptive lenses that guided the 
redesign process associated with the introduction of these tools. 

Descriptive Lenses Proposed by the Activity Theory Model 

Engestrom (1987) developed what is known today as the structure of human activity by focusing on the 
concept of mediation as the core element of the human activity. Evolutionary epistemology and the study of the 

genesis of intersubjectivity were the main complementary 
lineages of development for the concept of mediation used by 
Engestrom (1987, p.47). To convey the dynamic of a mediated 
act, Engestrom promotes the use of the triangle as the image of 
the basic mediated act (Figure 1). 

The proposed triangular representation conveys the 
complexity of mediated act as opposed to linearity of the dyadic 
link. On the other hand, the triangular representation is flexible 
enough to allow for the representation of complex human 
activities. The four major subsystems of the activity model 
presented in Figure 1, consumption, production, distribution and 
exchange provide a powerful descriptive tool for the analysis and 
design of instructional environments (e.g. Cernusca, 2008; 
Jonassen, 2000). However, for this study the analysis will cover 
only three of the subsystems: consumption, distribution and 
production. The focus of the analysis will be the production 
subsystem that describes the mediating role of the technology 
tools on the engagement of subjects (students and the instructor) 
on the object of the activity, the Hazardous Materials course. 




Figure 1. Human Activity Mediating Model 



Consumption and Distribution Subsystems: Impact on Technology Adoption Decisions 



The consumption and distribution subsystems cover the organizational level as they reflect to what degree 
the organization provides the motive, sustain and regulate the enactment of the analyzed activity. Analysis of 

consumption and distribution 
subsystems revealed factors that 
supported instructor’s decision to 
implement new technologies in his 
course. First, the university showed its 
commitment to technology enhanced 
effective learning environments both 
by including it in its mission and by 
committing resources to build the 
needed technology infrastructure. As 
shown in Figure 2, these efforts to 
build a culture that focuses on building 
student-centered learning apply to all 
instructional activities, including the 
STEM courses of which the Hazardous 
Materials course is part. They also 
address the needs and activities of both 
students and faculty members engaged 
in the instructional process at the 
university level. 




Figure 2. Consumption & Distribution Subsystems 
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Second, the distribution subsystem (Figure 2) indicates that the organization supports his mission through a 
flexible and effective division of labor for course design and redesign. That is, the university hired instructional 
technologists and designers that work closely with faculty members to find effective technology solutions for their 
courses. These factors strongly motivated the instructor in the Hazardous Materials course to continuously search for 
opportunities to enhance students’ learning experience in his course. The remaining part of this study describes such 
a step focusing on effective integration of technology-driven instructional tools and strategies in his course. 



Production Subsystem: Technology Selection and Implementation 



The production subsystem brings the analysis at the course level. As shown in Figure 3, analysis of 
production subsystem (subjects-tools-object) helped identify instructional activities that offer opportunities for 

enhanced outcomes. The production 




Figure 3. Production Subsystem 



subsystem contributes directly to the 
materialization of the outcomes through 
the mediated action of the subject on the 
object of activity by using specific tools 
and strategies. 

The analysis of existing 
situation at the time of this study 
indicated that the target course included 
both technology and non-technology 
driven active learning strategies. For 
example, students are engaged in small 
group discussions and peer-teaching 
activities throughout the course. Due to 
the complexity of the topic, response to 
hazardous materials incidents, the 
instructor also uses personal response 
systems (clickers) to pool ad-hoc 
opinions and use them to stimulate class- 
wide discussions. 



However, additional needs for two major instructional tasks emerged from the analysis of the production 
subsystem: the semester-long team project and the hands-on hazardous material incident response laboratories. 



Team Project. The semester-long team project has as main goals to help students: a) synthesize the 
knowledge covered in the course throughout the semester; b) apply these knowledge to a real-world scenario to 
build specific professional skills, and c) build teamwork collaborative skills specific to their professional field. The 
milestone reports and the final formal presentation addressed the first two of these goals but failed to provide 
students with an adequate platform to build collaborative skills. That is, group meetings and the use of email as 
communication tool for group project typically stimulate cooperation (split, do, combine to generate the final 
artifact) rather than collaboration (split, do, and contribute to the final artifact). These assessment tools also proved 
relatively weak in allowing the instructor to monitor and reward students for achieving this goal. To address this 
need the instructor and the instructional designer analyzed the features a Wiki space could offer for the term group 
project. 

As collaborative tools, Web 2.0 applications allow for the joint development of content and the unlimited 
sharing of information. They may also stimulate learners to get involved with their own construction of knowledge 
(Sigala, 2007). Wikis are asynchronous collaborative authoring tools that allow users, working either as individuals 
or in groups, to add and edit web pages, monitor changes, and discuss and negotiate emerging issues. Despite its 
relatively new presence in the educational landscape. Wikis were already adopted across various instructional areas 
such as computer science (Shih, Tseng & Yang, 2008), information systems (Ravid, Kalman, & Rafaeli, 2008). 
marketing (Cronin, 2009), management (Kosonen & Kianto, 2009), teacher education (Nicholas & Ng, 2009; 
Wheeler & Wheeler, 2009) or technical communications (Walsh, 2010). From a students’ perspective. Wikis have 
been previously used in classrooms as tools to document research projects (Engstrom & Jewett, 2005), increase the 
effectiveness of collaborative authoring (Bold, 2006; Ravid, Kalman, & Rafaeli, 2008 ), and support students’ 
engagement (Cole, 2009). From an instructor’s perspective, a major strength of this tool is its ability to allow the 
evaluation of individual contributions in a student collaborative activity (Trentin, 2009). Educators have access to 
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either free Wiki tools such as Google Sites ®, Wiki Spaces ® or PBWorks ® or proprietary Wiki tools such as 
Learning Objects ®. 

To address this need the instructor implemented Learning Spaces, a secure Wiki environment hosted in the 
Learning Management Systems (Blackboard) as the collaborative authoring and presentation tool for the term group 
project. To introduce students to this new tool, the instructor set up a face-to-face short session to present the tool 
and its features and a warm-up Wiki task that required each team to introduce their members in the group Wiki 
space. To stimulate individual responsibility as part of the teamwork, the instructor set up rotating roles (developer, 
editor, or reviewer) and allocated specific team participation points to each of them. 

Incident Response Laboratories . The two weekend-long laboratory activities are coordinated by active 
professionals and allow students to understand what it means to be a “first responder” to a hazardous materials 
incident. Therefore, these laboratories requires a significant investment from both the instructor, that has to secure 
the implication of the two professionals that coordinate these tasks, and from students, that need to commit out-of- 
class time for two full weekends. The analysis of this activity indicated one major weakness to be address. That is 
because of the scheduling issues these laboratories cover topics that are not covered in the lectures and homework 
activities and because of its process-oriented nature there is no artifact that can help students review this activity 
when these new topics are covered later in the course. 

The decision was to complement the two hands-on weekend activities with reflective discussion board 
forums. The implementation of this tool allowed the instructor to add critical reflection activities at the end of each 
hands-on activity. This addition provided also a scaffold for students’ learning as rational transformation through the 
integration new experiences with existing ones (Mezirow, 2000). In addition, these discussion forums: a) provide an 
artifact (archived discussions threads) that can help students recall major steps in each task, and b) create the 
opportunity for the instructor to measure the engagement and learning experiences of each student, beyond the 
observed physical participation in these activities. To ensure the effectiveness of these discussion forums, the 
instructor provided several seed questions and each student was required to answer one of these seed questions and 
reply to at least two threads posted by their colleagues. 

Because these interventions were implemented for the first time, the focus of this exploratory study was to 
identify how they are perceived by the students and if there are factors that can contribute to students’ resistance to 
these new instructional tools. 



Research Questions 

The exploratory questions for which this study seeks answers were: 

(1) Do students’ perceived engagement and difficulty reflect the potential impact of the included tools and 
associated instructional tasks? 

(2) Is there a difference between perceived engagement and difficulty of lecture and wiki tasks? 

(3) What is the students’ perception on discussion forums following the hands-on incident response activities? 

Research Methodology 

A number of 30 students from the Hazardous Materials Management course, mostly seniors, participated in this 
study. To measure the impact of these tools we used a mixed-methods approach. First, a set of nine -point 
differential-scale items (e.g. useless/useful, dull/lively, worthless/valuable, easy/hard) measured students’ perception 
on course engagement and difficulty (Bham et ah, 2010). For both engagement and difficulty the nine -point scale 
ranged from 5 (left) to 1 (comparison item) and to 5 (right) as shown in the sample item from engagement below. 
The middle of the scale, the comparison item, was “other courses” when the overall course was the subject of 
evaluation and respectively “lectures” when the Wiki was the subject of evaluation. 

other courses 

Boring 543212345 Interesting 

However, for the final analysis this scale was converted into a range from 1 (for low) to 9 (for high) as 
shown below in the recoded engagement item. 

other courses 

Boring 123456789 Interesting 

At the beginning of the class, these items were administered to record students’ expectation on course’s 
engagement and difficulty. 
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At the end of the course, the same items were administered measure perceived engagement and difficulty 
first related to the target course when compared to other courses students took at the same university, and second 
related to the wiki tasks when compared to the course overall. 

Finally, a series of open-ended questions were used to support a qualitative analysis of students’ 
perceptions on the impact of Wiki and respectively discussion forums on their learning experiences. Both the entry 
and exit surveys were administered online and rewarded with course participation points. 

Results and Interpretation 

A paired-samples t-Test indicated a significant difference between expected entry level and exit perceived level 
for both engagement and difficulty as follows. For the course, overall, the perceived engagement was significantly 
higher than the expected engagement (E entry = 7.37, SD = 1.04; E exit = 7.99, SD = .72), t(29) = -3.90, p < .01. In 
contrast, the perceived difficulty of the course overall was significantly smaller than the expected difficulty (D entry = 
5.19, SD = .77; D exlt = 3.58, SD = 1.47), t(29) = 5.23, p <.001. 

When the exit perceptions on the course and Wiki were compared to the middle of the scale “5”, an 
independent-samples t-provided some complementary information. For the course, when the middle of the scale 
represented other courses taken at the same institution, the perceived engagement for the target course was 
significantly higher than for other courses, t(29) = 22.66, p < .001. Following the previously found pattern of inverse 
relationship between the change in the difficulty and engagement, the perceived difficulty of the course was 
significantly lower than the perceived difficulty of other courses, t(29) = -5.27, p < .001. For the Wiki, when the 
middle of the scale represented the lecture, the perceived engagement was significantly higher for the Wiki than the 
lecture, t(29) = 4.68, p < .001. However, there was no statistically significant difference between the perceived 
difficulty of the Wiki (D wiki = 4.95, SD = 1.76) and lectures. 

These findings are well complemented by students’ answers on the open-ended questions related to both Wiki 
and discussion forums. For example, when directly asked if and why the Wiki made the project more engaging or 
not, 23 students (76.7%) answered positively and provided various arguments such as: 

Value for reporting: “If 1 had to chose between the wiki or a report the wiki beats report writing by a 
mile.” (Student 1) 

Convenience and easiness of use: “Not only was it a great interface for us to work together while being in 
separate place, but it is also a great to for making a webpage” (Student 5), or 

A good tool for organizing the content: “. . . We had to think about the presentation of the writing in a way 
that is different from all other classes and 1 liked it.” (Student 9) 

However 2 students (6.7%) found the Wiki moderately useful while 5 (16.6%) had a negative opinion about 
this tool either because they found it less engaging than face-to-face meeting they missed by using Wiki or because 
they felt they still had to work more than their colleagues to make sure the final grade will met their expectation. 

When the students were asked about the usefulness of the discussion forums following the hands-on activities, 
slightly more than half of them 16 (53.3) provided strong supporting answers, while 8 (26.7) considered them 
marginally useful and 6 (20%) find them not useful. Most of the complaints were related to the redundancy of this 
task as each activity was followed by short debriefing while others complained about the additional load created by 
the need to read others’ postings. Overall, therefore students’ input through their answers to the open-ended answers 
supported the increase in engagement and decrease in difficulty reflected by the results from the quantitative 
analysis. 



Conclusions and Further Research 

The positive findings revealed by students’ perceptions indicate that integrating in a meaningful way 
technology-driven instructional tools that complement active learning strategies can enhance overall learning 
experience for most students. Students also provided valuable insights on areas that can benefit from further 
improvements. To strengthen these findings, future research will be conducted to: a) extend the engagement and 
difficulty scales to the discussion forums, and b) develop evaluation tools (e.g. rubrics) to link students’ perceptions 
to their performance in both Wikis and reflective discussion forums. 
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1. Introduction 

With the great advance of the information technologies and new educational policies for building 
a better learning environment, "information technology integrated teaching" and "adaptive teaching" 
have become the important tasks to be completed for all-level schools. After the installation of 
hardware and completion of networking, it is the key for educational reformation that using innovative 
information immersion education to enhance the quality of teaching and to provide a learning process 
suitable for students. 

During current fast moving e-leaming era, setting up a qualified e-leaning system to improve 
students’ ability has become the most important issue in higher education. At the same time, along with 
the cutting-edge technologies upgrades, the new learning approach, blended learning, has been 
developed, which actually has been the most popular learning trend these days in Taiwan. 

A blended learning environment combines face to face classroom methods with 
computer-mediated activities to form an integrated instructional approach. In the past, digital materials 
have served in a supplementary role, helping to support face to face instruction. This provides the 
learner based multiple learning methods to education instructors and also break the traditional 
classroom education boundaries with unlimited ways to independent learning. Moreover, it can also 
combine educational technology and ubiquitous learning to achieve the maximum learning 
effectiveness. 

Blended learning is also the process of incorporating many different learning styles that can be 
accomplished through the learners’ blended virtual and physical resources. Many scholars mentioned 
that with blended learning and lessons arrangements combining digital education and actual lessons to 
solve the problems as human interaction and sense of participation can be the best way to maximize the 
e-leaming benefits. 

In Taiwan, blended learning is on the rise in higher education, many higher education instructors 
and administrators are using blended learning strategies somewhere in their institution. Some 
universities are expecting more than 50% of their university courses to be blended. 
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These trends emphasize that learning is becoming more social and infonnal and less structured. 
Finding activities in understanding the student experience in blended learning is crucial important. 
Developing a realistic, detailed sense of the students’ learning styles is also an important starting point 
to the design process. 



The goal of a blended learning is to join the best aspects of both face to face and online 
instruction. Classroom time can be used to engage students in advanced interactive experiences. 
Meanwhile, the online portion of the course can provide students with multimedia-rich content at 
anytime, anywhere the student has internet access, from university computer labs, the coffee shop, or 
the students’ dorms. This allows for an increase in scheduling flexibility for students. In addition to 
flexibility and convenience for students, there is early evidence that a blended instructional approach 
can result in learning outcome gains and increased enrollment retention. 



2. Measures 

This study was conducted in a university of north-eastern Taiwan, in 2010, for the purpose of 
discussing in the blended learning curriculum, the Engineering college students’ learning style 
distribution, and examined the gender, age, and learning achievement differences on each learning style 
dimension, by using questionnaire survey procedure, with Felder & Soloman Index of Learning Style, 
as the instrument. The sample size (valid survey) is 276 and the distributions by gender, dept., grade 
and achievement are as the following tables and figures. 
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The Felder & Silverman learning-styles emphasize that each person is unique, can learn, and has 
an individual learning style. Effective teachers continually monitor activities to ensure compatibility of 
instruction with each individual's learning-style strengths and the adaptive curriculum and instruction 
are learning-style based and personalized to address and honor diversity. During the past 30 years, 
extensive cross-subject research has been conducted with the Felder & Silverman learning-styles model 
in diverse counties on all continents of the world. Many of the research result and knowledge 
concerning instructional technology can be the reference for comparison while we are giving an 
impulse to "information technology integrated teaching" and "adaptive teaching". 



3. Data analysis and results 

The results demonstrated the Engineering college students’ learning style distribution condition, 
is the intense image visual type learner on the learning sense organ; the learning manner is partial to the 
reconsideration slightly; the learning way is partial to the feeling slightly; the learning pondered that 
the pattern is partial to the total build slightly (please see the following tables and figures). 
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This research also examined all students, students in various departments, and different age 
students by T-Test, it indicated there’s gender differences(p=.014<.05) in learning manner. 
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By single factor variance analytical control various departments, various grades, with the 
different learning achievement's learner, in four learning style dimensions, besides the different studies 
achievement's learner not remarkable difference, it showed there were significant differences in 
learning styles for students in different departments, in learning manner(F(3,272)=2.978’P=.032 < •05), 
and the different grades, in learning way(F(3,272)=3.929 > P=.009 < .05). Please see the following 
tables. 
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4. Discussion and conclusions 

Therefore, the aim of this study was to use the learning-styles model to promote technology 
integrated teaching and to assist students' learning in blending learning environment. By examining the 
students' learning-style strengths and characteristics by age, gender, department and achievement, here 
comes some concrete suggestions to improve the instruction in blended learning environment and 
enhance students' learning embedded with "information technology integrated teaching" and "adaptive 
teaching" and suggestions for further study. 
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Visual learners remember best what they see— pictures, diagrams, flow charts, time lines, films, 
and demonstrations. Verbal learners get more out of words— written and spoken explanations. Everyone 
leams more when infonnation is presented both visually and verbally. In most college classes very little 
visual infonnation is presented: students mainly listen to lectures and read material written on 
chalkboards and in textbooks and handouts. Evidence shows that the most Engineering students are 
visual learners, which means that most students will get nearly as much as they would if more visual 
presentation were used in blended class. Good learners are capable of processing infonnation presented 
either visually or verbally. 

Reflective learners prefer to think about the material first. It will be benefit from periodically 
reviewing what has been read and thinking of possible, questions and applications and writing a 
summary of readings or class notes. Evidence shows that the most Engineering students’ learning 
manner is partial to the reconsideration slightly. For instructor, if there are reflective learners in a class, 
these students have little or no class time for thinking about new information, it will be helpful to 
schedule time to reflect on material and remind students don’t just read, stop periodically to review the 
material and think of possible questions or applications, ask students to write short summaries of 
materials read, and use reflective writing tasks in blended class. 

Evidence shows that the most Engineering students’ learning way is partial to the feeling slightly. 
Sensing learners tend to like learning facts, instead of discovering possibilities and relationships. 
Sensors often like solving problems by well-established methods and dislike complications and 
surprises and resent being tested on material that has not been explicitly covered in class. Sensing 
learners also tend to be patient with details and good at memorizing facts and doing hands-on 
(laboratory) work and to be more practical and careful than intuitive learners. If the instructors can 
provide sensing learners the apparent connection to the real world and the "plug-and-chug" courses that 
involve a lot of memorization and routine calculations, this design will help a lot in blended class for 
Engineering students. 

Global learners may be able to solve complex problems quickly or put things together in novel 
ways once they have grasped the big picture, but they may have difficulty explaining how they did it. 
Global learners tend to learn in large jumps, absorbing material almost randomly without seeing 
connections, and then suddenly "getting it”. The research result shows that the Engineering students are 
more global learners. Strongly global learners who lack good sequential thinking abilities, on the other 
hand, may have serious difficulties until they have the big picture. Even after they have it, they may be 
fuzzy about the details of the subject, while sequential learners may know a lot about specific aspects 
of a subject but may have trouble relating them to different aspects of the same subject or to different 
subjects. If instructor plunges directly into new topics without bothering to explain how they relate to 
what learner already know, it can cause problems for “global” students in blended class. 
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Gender and age differences were also be found in this research. It will suggest that the future 
study to investigate the relationship between the intensity of the use of the blended learning 
environment and student background characteristics, such as learning style preferences, achievement 
motivation, self-concept constructs and subject attitudes. 
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Abstract 

This study employed the theory of planned behavior as a framework for identifying college students’ current 
perceptions and needs for mobile learning. The use of mobile devices continues to evolve, and many educators are 
eager to explore the potential of these devices to enhance student-centered learning by facilitating any time/any where 
collaboration and communication. Self-reported data from 238 college students was analyzed with a structural 
equation modeling method. The results confirmed the theoiy that their attitude, behavioral control and subjective 
norm positively influenced their acceptance of m-learning, while they perceived that a social environment is not 
strong enough to implement m-learning. In addition, other findings revealed preferable instructional activities with 
mobile devices in higher education. 



Introduction 

In recent years, we have witnessed an explosion in the growth of mobile devices, such as smart phones 
(e.g., iPhone) and mobile tablets (e.g., iPad) which use 3G or wireless networks. These devices are altering how we 
live and how we learn (Abdullah & Siraj, 2010). Mobile learning (m-learning) enables people to access learning 
anytime and anywhere. These devices are also important for supporting just-in-time, customized, and life-long 
education. Since college classrooms are filled with students living in a mobile age, institutions in higher education 
have an opportunity to revitalize the process of teaching and learning via m-learning. However, m-learning is still in 
its infancy in higher education. Many universities provide a free App (an application for a mobile phone), but it 
contains mostly non-instmctional contents such as news, event calendars or maps. Although m-learning has the 
potential to augment formal education with flexible access, immediate communication and supplemental learning 
materials, there are serious concerns about the readiness of college campuses to adopt m-learning (Al-Mushasha, 
2010), and there is lack of research exploring the readiness of college environments for m-learning. 

This study adapted the theory of planned behavior (TPB) to investigate the determinants of college 
students’ intention to use m-learning. The theory focuses on the formulation of an intention to behave in a particular 
way, and the sources of the intention are attitude, subjective norm, and behavioral control (Ajzen, 1991). Based on 
this approach, we proposed new antecedents of attitudinal constructs and draw out conceptual frameworks. Our 
research questions were: (a) What are the significant salient beliefs of college students that contribute to the levels of 
attitudinal constructs? (b) How strongly do their attitudinal constructs influence their intention to use m-learning? (c) 
How do college students want to use a mobile device in their course work? The answers to these questions will 
allow us to identify the readiness of college students for m-learning which will be a basis for designing effective m- 
learning environments in higher education. 



Mobile learning 

m-learning refers to any sort of learning that happens when the learner is not at a fixed, predetermined 
location, or learning that happens when the learner takes advantage of learning opportunities offered by mobile 
technologies (O’Mally et al., 2003). m-learning, in general, could enrich students’ learning experiences with 
enhanced mobility and connectivity. More specifically, there are five educational benefits based on previous 
literature: (a) portability, (b) interactivity, (c) context sensitivity, (d) connectivity, and (e) individuality (e.g., 
BenMoussa, 2003; Churchill & Churchill, 2008; Sharpies, 2000) . Previous research has proposed general 
considerations for m-learning (e.g., Gu, Gu & Laffey, 2011; Liu, Li & Carlsson, 2010; Shih & Mills, 2007). For 
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example, Gu et al. argued that m-learning contents for life-long education should address practical, real life needs; 
be micro in terms of length and time; and be simple and easy to understand. In addition, they discovered that audio 
is the preferred modality, and that usability requirements should be consistent with those required for web pages 
(i.e., minimizing scrolling). Liu, Li and Carlsson (2010) found that perceived usefulness and personal innovation 
significantly influence the intention to adopt m-learning. On the other hand, based on Keller’s ARCS model, Shih 
and Mills (2007) suggested five steps to increase learners’ motivation with mobile activities. Other studies 
investigated learners’ perceptions toward educational benefits of m-learning (e.g., Abdullah & Siraj, 2010; Al- 
Mushasha, 2010), and optimistic results were found. 

Research Model and Hypothesis Development 



Theory of Planned Behavior 

The theory of planned behavior (TPB) was selected as a research model in this study. The TPB explains 
attitudinal factors of an individual’s intention to behave in particular way. In the TPB’s framework, intention is 
predicted by three determinants: (a) attitude, (b) subjective norm, and (c) perceived behavioral control (Ajzen, 1985, 
1991). Many studies have applied this theory in different contexts (e.g., technology, health care, and political 
science) or with different antecedent variables of the attitudinal construct in order to predict behavior (e.g., Conner 
& Armitage, 1998; Davis, 1989; Taylor & Todd, 1995). Based on the TPB, we formulated our research model as 
shown in Figure 1. We used the behavioral intention as an ultimate perception in this study as the TPB proposed. 

The research model proposes that behavioral beliefs reinforce three attitudinal constructs, and, in turn, the constructs 
positively enhances behavioral intention. The antecedent variables of each attitudinal construct shown in the left 
column are proposed from various theories, and all hypotheses are described below. 

Attitudinal Constructs and Behavioral Intention 

First, attitude toward behavior refers to the degree to which a person has a favorable or unfavorable feeling 
about performing a particular behavior. Previous studies found that attitude is a strong predictor of intention (Davis, 
1989; Taylor & Todd, 1995). Second, subject norm is about a social environment. In other words, an individual 
integrates others’ opinions into his/her belief and performs a similar behavior to others (Venkatesh & Davis, 2000). 
Last, regarding behavioral control, he/she perceives greater control, which triggers an intention to perform the 
behavior, when an individual perceives that he/she has more resources and confidence than expected obstacles 
(Ajzen, 1985; Hartwick & Barki, 1994; Lee & Kozar, 2005). Therefore, we hypothesized: 

• HI: College students’ attitude toward m-learning positively influences their intentions to use m-learning. 

• H2: College students’ subjective norm of m-learning positively influences their intentions to use m-learning. 

• H3: College students’ perceived behavioral control of m-learning positively influences their intentions to 
use m-learning. 
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Attitudinal Beliefs toward Attitude 

The antecedents of the first attitudinal construct (i.e., attitude) are attitudinal beliefs. In our research model, 
variables for attitudinal beliefs are derived from the technology acceptance model (TAM) which explains how 
people accept a new system. It argues that perceived ease of use and usefulness determine an individual's intention 
to use a system (Davis, 1989). Also, perceived usefulness is affected by perceived ease of use. Accordingly, we 
include the two perceptions in our belief constructs and hypotheses, 

• H4: College students’ perceived ease of use of m-learning positively influences their perceived usefulness 
of m-learning. 

• H5: College students’ perceived ease of use of m-learning positively influences their attitude toward m- 
learning. 

Normative beliefs toward Subjective Norm 

Subjective norm is determined by the accessible normative beliefs that accounts for social pressure from 
referent as an important determinant in an individual’s behavioral intention. Since individuals are dependent on 
context, and they are socially constructed beings (Shah, 1998), we propose that other people in their academic life 
(i.e., instructor and other students) can affect the subjective norm of m-learning for college students. Thus, we 
hypothesized: 

• H6: Perceived instructors’ readiness of m-learning positively influences their subjective norm with m- 
learning. 
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• H7: perceived other students’ readiness of m-learning positively influences their subjective norm with m- 
learning. 

Control beliefs toward Perceived Behavioral Control 

Perceived behavioral control is compatible with the concept of self-efficacy. In other words, individual’s 
confidence in performing a specific task significantly influences behavior (Ajzen, 1991). Self-efficacy refers to 
individuals' beliefs about their ability and motivation to perform specific tasks (Bandura, 1986, 1997). In other 
words, individuals who believe they can master a certain skill or an activity tend to have higher intention to obtain 
the skill or perform the activity. Previous studies found that higher levels of self-efficacy will lead to higher levels of 
behavioral intention and the usage of information technology (Compeau & Higgins, 1995; Gist, Schwoerer, & 

Rosen, 1989). In addition, this study employed learning autonomy as the second antecedent. Learner’s autonomy 
toward m-learning is whether they can control the learning pace and style of interaction. Autonomy has proved to 
be a major contributor to system acceptance (Liaw, Huang, & Chen, 2007). Therefore, we hypothesized: 

• H8: College students’ perceived self-efficacy of m-learning positively influences their behavioral control 
with m-learning. 

• H9: College students’ perceived learning autonomy of m-learning positively influences their behavioral 
control with m-learning. 



Method 

To address the above research questions and investigate our hypotheses, we collected data from college 
students using a survey instrument adapted from previous studies. The survey data was analyzed to test the 
hypotheses with Partial Least Squares (PLS) Graph. The participants of this research were 238 undergraduate 
students at a large university in the southwestern United States (Male: 114, Female: 124). One hundred and eighty 
students had a smartphone (iPhone: 111, other smartphones: 69). The most frequent use of their phones was texting 
followed by accessing social networking services (i.e., Facebook or Twitter). All data about their mobile phone use 
will be presented at the conference. We developed the survey instrument containing 30 items (three items for 10 
constructs). The survey measured participants’ perceptions with 7-point Likert scales ranging from totally disagree 
to totally agree. In addition, preferable learning activities with mobile devices were collected by six items with 7- 
point Likert scales as well. 



Results 

Confirmatory factor analysis was conducted to assess the measurement scales’ validity using PLS-Graph, 
version 3.0. Table 2 shows that the composite reliability for all constructs is greater than 0.80 and the average 
variance extracted (AVE) is greater than 0.50. Also, all item-loadings were greater than 0.70; therefore, the level is 
generally acceptable (Fornell & Larcker, 1981). In this study, AVE for each construct is greater than the correlation 
between that and all other constructs. The structural model analyzed the relationships between the various latent 
variables. Figure 2 presents the standardized path coefficients and the explained construct variances. 
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* p < .05, ** p < .01 



Figure 2. Results of the structural model 



The results show that all hypotheses were supported. Hypotheses 1 to 3 were supported (HI, coefficient of 
0.48, t-value of 7.33, p < 0.01; H2, coefficient of 0.16, t-value of 2.87, p < 0.01; H3, coefficient of 0.37, t-value of 
5.1 1, p < 0.01). In other words, behavioral intention was positively influenced by attitude, subjective norm and 
perceived behavioral control. However, the effect of subjective norm was less than other constructs. All hypotheses 
regarding the relationships between three attitudinal constructs and antecedent variables for each construct were 
supported. First, both perceived ease of use (H4, coefficient of 0.21, t-value of 3.04, p < 0.01) and perceived 
usefulness (H5, coefficient of 0.69, t-value of 11.23, p < 0.01) made a significant effect on attitude. Second, the 
assumption of the positive relationship between the readiness of instructor and other students and subjective norm 
was met (H6, coefficient of 0.41, t-value of 6.38, p < 0.01; H7, coefficient of 0.41, t-value of 5.96, p < 0.01). Last, 
we found that both hypotheses 8 and 9 were supported. For example, perceived self-efficacy positively influenced 
perceived behavioral control (H8, coefficient of 0.62, t-value of 7.48, p < 0.01). Furthermore, learning autonomy 
favorably influenced perceived behavioral control (H9, coefficient of 0.23, t-value of 2.74, p < 0.01). 

In addition, a dependent /-test revealed that the perceived readiness of students (M= 5.56) was significantly 
higher than the readiness of instructors (M = 4.67, i(237) = 12.60, p < .001). The results of another survey showed 
that accessing course information (e.g., schedulers, exam results,) (M = 6.07) was the most highest activity 
participants want to do with their mobile devices. The second one was communication with instructors ( M= 5.77). 

Discussion and Conclusions 

In sum, this study allows us to empirically investigate the effects of college students’ perceptions toward 
m-leaning intention. First, the significant impact of perceived ease of use and usefulness on attitude confirm the 
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technology acceptance model (TAM). In other words, college students who feel that m-learning is easy to use and 
useful are more likely to use mobile devices for their course work. Since the coefficient value of perceived 
usefulness (0.69) was higher than the value of perceived ease of use (0.21), it can be said that the perception toward 
usefulness is a stronger contributor to attitude. We assume that they were already familiar with mobile devices, and 
the advantages of using the mobile devices for the courses highly influenced the attitude toward m-learning. 

Second, the results showed that college students’ behavioral control was another important facilitator of 
their intention to use m-learning. Although both antecedents positively affect the behavioral control, their self- 
efficacy (i.e., confidence, Agarwal & Karahanna, 2000) had higher effect on the perceptions of control over m- 
learning. This finding implies that empowering students’ with confidence in using m-learning will lead to a greater 
likelihood of technology adoption. 

Third, although a significant relationship between subjective norm and intention was found, the effect was 
somewhat lower than other two constructs. This finding is consistent with what Shiue (2007) found in which 
subjective environment weakly influenced the actual use of technology. Interestingly, college students’ thought that 
their instructors may not be ready to use mobile devices in their courses. 

Last, the first and second highest favorable activities with mobile devices were accessing course 
information and instructors. In order to make the course syllabus or schedule available for mobile devices, course 
website or learning management system should be mobile friendly. Thus, institutional support is necessary to 
implement the mobile supports. Another support for faculty members is also required, such as professional 
development, online space with real examples, or learning communities. 

This study has a number of limitations that circumscribe our interpretation and create opportunities for 
future research. Since the participants watched three video clips that showed some examples of m-learning, they 
might have favorable bias toward m-learning. Furthermore, they have not had a chance to utilize mobile learning for 
their course works except exchanging email, and the learning management system in the university does not have a 
mobile application (i.e., App). Thus their responses were not derived from a real situation. However, the results 
showed that the students are highly favorable toward using mobile devices for their learning. This study may be 
extended to college faculty members to compare their perceptions to student’s perceptions. Our findings show that 
emphasis on the three elements for college students’ perceptions could enhance their actual adoption of new 
technologies which change the way they learn. Detailed discussion from this study will be presented at the 
conference. 
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Abstract 

This study is interested in information age qualities of principles, teachers and students in Turkish Vocational High 
Schools. A readiness to systemic change survey was used as the measuring tool for the information age qualities. 
The survey was implemented to 7 schools and there was a total participation of 29 teachers and 282 students. The 
data obtained from the administration of the measuring instalment was analyzed by using both descriptive and 
inferential statistics. Findings of the readiness survey indicated that teachers’ perceptions and relationship with the 
principal mean scores are higher than students. When the data were analyzed using Analysis of Variance between 
groups (ANOVA), the results indicate that there were significant differences in how the two stakeholder groups 
perceives principals, teachers and students information-age quality measures. 

Introduction 

Turkey is a fast developing country in the edge of passing from the societal and educational needs of the 
industrial age to the needs of information age. The more the society and jobs rely on information and information 
processing, the needs of the Turkish educational system increases and changes rapidly. This is a serious situation 
that needs to be taken into account since societies still valuing industrial age qualities expect that jobs need manual 
labor. But the current information age jobs require knowledge work based on solid information age qualities and . 
While during the industrial age, a comfortable middle-class life was possible without much education, whereas in 
this age of global competition and digital technologies, considerably higher levels of education are needed to have a 
comfortable life. As Turkey becomes more and more close to information age, the business, societal and educational 
needs are getting more and more complex. The way we work, the way we learn, the way we play are becoming more 
and more complex every day. Even the way we socialize by using the new tools of the Internet and web 
technologies are becoming more complex. It is interesting that the means and solutions to handle this complexities 
by using technology-based tools create further complex job areas. Spiro (2006) refers to these new technologies as 
“Post-Gutenberg technologies” and points on the need for drastic changes in goals and means of education for the 
development of a different style of thinking, through “prefigurative schemas” (schemas for the development of 
schemas) so as to deal with this complexity. 

Solutions provided in Turkish educational system to meet the needs in Industry in terms of workforce that 
can deal with the complexity of the workplace knowledge needs to be carried out for all times. General approach is 
to focus on the measurement, evaluation and placement issues. There have been changes in the way how students 
are measured, evaluated and placed on to educational tracks starting from elementary school and going up to post- 
graduate levels. Focusing on the examination is a narrow view to the problem of quality of education. The solution 
for the growing low educational quality crisis is hard to solve with piecemeal change. Making changes in the 
curriculum by increasing or decreasing class hours of some courses does not add to the solution but even creates 
more dramatic problems such as placement of students and teachers to the educational programs. What we need to 
recognize is that dramatic changes in educational needs require changes in the fundamental structure and 
organization of schools. Schlechty points on the need to reconsider the “rules, roles, and relationships” for the ways 
we use “time, talent, and technology” in schools (Schlechty, 1990). A typical example for this is the way we use 
time in our educational system. When the curriculum is developed or renewed we require all students to leam the 
same amount of content in the same amount of time. On the other hand, we know that students do not learn at the 
same pace. 

Turkey, with a high population of young people needs to educate more and more students that can 
potentially deal with the knowledge requirements of the information age. In the current system students have to go 
through classes based on pre-determined time periods. The use of time has to be reconsidered and redesigned so it is 
not a typical constant of students’ achievement anymore. A dramatic change as referred by Schlechty (1990) would 
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to redesign schools without class periods and grade levels. This type of change would also affect teachers and 
students qualities in learning and communication and the use of instructional technologies in the schools (Schlechty, 
2002). As it is stated by Reigeluth (1999), the new structure would require fundamental shifts in structure and 
qualities like from standardization to customization, from control to empowerment, from compliance to initiative, 
and from uniformity to diversity. 

Purpose of the Study 

It is important to build new knowledge about how to start a successful systemic transformation process. 

The information age qualities investigated of the principals, teachers and students as evaluated by teachers and 
students will provide an initial measure of how ready Turkish vocational schools are to a dramatic systemic change. 
The study has two purposes. The first is to descriptively present the current information age qualities of principles, 
teacher, and students. The second purpose is to examine if there are statistically significant mean differences 
between the students and teachers in perceived information age qualities. 



The research questions that guide this study are listed below: 

1. What are the perceived information age qualities of principals, students and teachers in Turkish vocational 
schools? 

2. Are there any differences in the perceived information-age qualities between the teachers and the students? 

Sub Questions: Are there any differences in perceived: 

2.1 Principals information-age qualities? 

2.2. Teacher Information- Age qualities? 

2.3. Relationships with the principal? 

2.4. Student readiness for assuming responsibility for their own learning? 

Methods 

To answer the research a question, a survey research was conducted in 7 vocational schools. A total of 29 
Teachers and 282 students in Turkey participated to the study. 

Design of the Study 

This study used a survey research design, along with inferential statistics to compare readiness levels of 
students and teachers in vocational schools. There were one independent variable (IV) and 5 dependent variables 
(DVs). The DVs are divided into two groups: Information-Age Qualities and Relationships. Table 1 summarizes the 
characteristics of these variables. 



Table 1- Identification of the Variables 



TYPE OF VARIABLE 


NAME 


TYPE OF VALUE 


TYPE OF SCALE 


IV 


Stakeholder Group 
(Teacher and 
Students) 


Discrete 


Nominal 


Information Age Qualities 


DV 


Principal Qualities 


Continuous 


Interval 


DV 


Teacher Qualities 


Continuous 


Interval 


DV 


Student Readiness 


Continuous 


Interval 


Relationships 


DV 


Teachers and Students 
with Principal 


Continuous 


Interval 


DV 


Students with 
Teachers 


Continuous 


Interval 
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Survey Instruments 



For this study, two different versions of one measure, the Readiness Survey, were used. These versions were 
the (1) Readiness Survey for Teachers and the (2) Readiness Survey for Students. The surveys were adapted from an 
earlier version that was developed by the systemic change research group in Indiana University to be used in Decatur 
district Systemic change efforts guided by Richter and Reigeluth (2006). Table 2 shows the items that were included 
in each survey. There were minor differences among the two versions of the survey, taking into account the 
demographic differences between the groups. There were five dimensions in each version of the survey. The 
response scale for the items in each dimension was a 5-point likert scale. The dimensions were: 

• Principal’s Information- Age Qualities 

• Teacher Information- Age Qualities 

• Relationship with the Principle 

• Relationship with the Teachers 

• Student Readiness for Assuming Responsibility for their Own Learning 



Table 2- Items Included in Different Versions of the Readiness Survey 





Teachers 


Students 


Principal’s Qualities 






Style of leadership 


V 


V 


Mindset about education 


V 


V 


Willingness to try new ideas 


V 


V 


Ability to inspire people through adversity 


V 


V 


Teachers’ Qualities 






Mindset about education 


V 


V 


Experience with learner-centered instruction 


V 




Willingness to try new ideas 


V 


V 


Desire for innovation 


V 


V 


Relationships with Principal 






Trust 


V 


V 


Respect 


V 


V 


Cooperation 


V 


V 


Collaboration 


V 


V 


Relationships with the Teachers 






Trust 




V 


Respect 




V 


Cooperation 




V 


Collaboration 




V 


Student Readiness 






Readiness of your students to assume more responsibility for their own 
learning 


V 


V 



Reliability: Reliability of the survey instrument is based on the internal consistency of the items. The internal 
reliability coefficient was obtained for each readiness dimension and for the overall survey by using the Cronbach 
alpha coefficient. Readiness Survey for Teachers and the Readiness Survey for Students have a high level of 
reliability (0.83 and 0.82 respectively). 

Validity: In research studies where data are collected via surveys, validity is a big concern. In this study, the data 
collection instrument went through a detailed development phase. For validity concerns, the readiness surveys were 
adapted from an earlier version that was developed by the systemic change research group in Indiana University in 
Decatur Systemic change efforts of Richter and Reigeluth (2006). 



52 





Participants and Context 



The participants of the study consisted of 29 teachers and 282 students responding to the surveys. It is 
important to point that the study had the normal limitations of a survey research in that only present results for 
teachers and students who chose to participate in the survey could be included. The surveys were administered 
during 2008-2009 school year to 7 vocational schools through the Internet. Teacher from technical branches of the 4 
different vocational school types were requested to fill the surveys. The school types that were included in the study 
are explained below; 

1. Vocational High Schools: Intends to provide the labor force in the Industry. The primary aim is not to 
prepare students for higher education but to join the intermediary work force. These students are offered 
limited number of science and math courses. 

2. Anatolian Vocational High Schools: Intends the same as the vocational schools but the student 
chooses his/her branch at the beginning in the entry exam made for these types of schools. The students 
attend a 1 year prep-school to learn a foreign language (majority English, a few German and French 
offerings are available) before starting their 9th grade. These schools offer limited number of science and 
math courses. 

3. Technical High Schools: Intends to provide pre-skilled students to higher education programs or as the 
intermediary work force to the industry. Students take the science and math course trough out their four 
year of school grades to have a chance in the University entry exam. At the same time the student take 
vocation oriented courses. The vocational courses are the same as the vocational schools students take. 

4. Anatolian Technical High Schools: Is the same as the Technical schools except that the students are 
accepted trough the Anatolian High schools entry exam and know their department right from the 
beginning. The students have to attend the prep-school for one school year to learn English before passing 
to the 9th grade. 



Teachers were contacted trough phone calls and e-mails and encouraged to participate in the study. Those 
who agreed to participate were requested to ask their students for participating to the study and were provided by 
survey web-site address. The survey taking date of the students was scheduled for date and time. Teachers were 
reminded to take their student to computer labs on the date of student survey dates. 

Data Analysis Procedures 

In order to answer the research questions of the study, two data analysis phases were conducted. The first 
phase was descriptive statistics, including means, and standard deviations. In this phase, the first research question 
was addressed. 

The second phase used inferential statistics and addressed the second research questions and sub-questions. 
ANOVA was used to test for statistically significant differences in readiness levels between teachers and students. 

Results 

The results are provided separately for each research question of the study. In the first part, results on the 
descriptive research question are presented. Second, results on ANOVA are presented. These results present the 
differences in the information-age dimensions between teacher and student perceptions. 

Descriptive Results 

The mean scores for information-age qualities of principals, teachers and students are presented in Table 3. 
The response scale for the items in each dimension was a 5-point likert scale. Teachers and students evaluated the 
information age qualities of themselves and of others. While all of the teachers corresponded to all items in the 
survey, some students did prefer to not respond to some items. The mean score that is higher in comparison is 
underlined. 
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Table 3- Means of Perceived Information Age Qualities of Principals, Teachers and Students 







Teachers 




Students 




N1 


Mean 1 


Std. Dev. 1 


N2 


Mean 2 


Std. Dev. 2 


Principal’s Qualities 


29 


3.72 


0.59 


263 


3.48 


0.97 


Style of leadership 


29 


3.62 


1.02 


270 


3.44 


1.18 


Mindset about education 


29 


3.62 


0.94 


275 


3.68 


1.16 


Willingness to try new ideas 


29 


4.45 


0.69 


279 


3.48 


1.22 


Ability to inspire people through 


29 


3.21 


0.82 


275 


3.32 


1.25 


adversity 

Teachers’ Qualities 


29 


3.99 


0.58 


269 


3.58 


0.95 


Mindset about education 


29 


4.00 


0.60 


279 


3.78 


1.11 


Experience with learner- 
centered instruction 


29 


3.76 


0.87 




Willingness to try new ideas 


29 


4.24 


0.64 


278 


3.49 


1.15 


Desire for innovation 


29 


4.21 


0.68 


272 


3.47 


1.17 


Relationships with Principal 


29 


3.92 


0.82 


276 


3.49 


1.20 


Trust 


29 


3.90 


0.98 


281 


3.33 


1.46 


Respect 


29 


3.72 


0.10 


281 


3.89 


1.34 


Cooperation 


29 


4.34 


0.86 


280 


3.68 


1.43 


Collaboration 


29 


3.69 


1.10 


279 


3.07 


1.44 


Relationships with the 
Teachers 








282 


3.94 


1.02 


Trust 








279 


3.91 


1.25 


Respect 








279 


4.13 


1.12 


Cooperation 








282 


4.00 


1.13 


Collaboration 








244 


3.67 


1.26 


Student Readiness 

Readiness of students to assume 
more responsibility for their own 
learning 


29 


2.34 


0.94 


274 


4.04 


1.01 
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Differences in the Perceived Information-age Qualities between Teachers and Students 



The scores for the readiness dimensions common in both surveys, that is, the teachers and student versions, were 
compared and the results are presented below. ANOVA was run to understand if there were significant differences 
in the following dimensions between teacher and student perceptions: 

1. Principals’ information-Age qualities 

2. Teacher s’ Information-Age qualities 

3. Relationships with the principal 

4. Student readiness for assuming responsibility for their own learning 

The results of the ANOVA on principles information-age qualities are presented in Table 4. 

Table 4- ANOVA Results for perceived principal qualities 







df 


F 


Sig. 


Principal’s Information-age 


Between Groups 


1 


1.765 


0.185 


Qualities 


Within Groups 


290 








Total 


291 






Principal’s Leadership Style 


Between Groups 


1 


0.624 


0.430 




Within Groups 


297 








Total 


298 






Principal’s Mindset about 


Between Groups 


1 


0.070 


0.791 


Education 


Within Groups 


302 








Total 


303 






Principal’s Willingness to tiy 


Between Groups 


1 


17.730 


0.000* 


New Ideas 


Within Groups 


306 








Total 


307 






Principal’s Ability to inspire 


Between Groups 


1 


0.240 


0.624 


People through Adversity 


Within Groups 


302 








Total 


303 







* p<0.05 



As shown in Table 4, there is no significant difference in overall, but significant differences existed in only 
one of the measures between teachers and students perceptions on principals’ information-age qualities. Teachers 
perceptions on “Principal’s Willingness to try New Ideas” was significantly higher, whereas Leadership style, 
Mindset about Education, and Ability to inspire People through Adversity did not show any significant differences 
between the two stakeholder groups. The results of the ANOVA on teachers’ information-age qualities are 
presented in Table 5. 



Table 5- ANOVA Results for teachers’ information-age qualities 







df 


F 


Sig. 


Teachers’ Information-age 


Between Groups 


1 


5.094 


0.025* 


Qualities 


Within Groups 


296 








Total 


297 






Mindset about education 


Between Groups 


1 


1.053 


0.306 




Within Groups 


306 








Total 


307 






Willingness to try new ideas 


Between Groups 


1 


12.143 


0.001* 




Within Groups 


305 








Total 


306 






Desire for innovation 


Between Groups 


1 


10.879 


0.001* 




Within Groups 


299 








Total 


300 







* p<0.05 
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As shown in Table 5, another significant finding of the study was the difference between perceived 
information-age qualities of the teachers’. Teachers perceived their information-age qualities significantly higher 
than how students perceived them. All measures about teacher information- age qualities included in the survey, 
except mindset about education, was perceived by the teachers significantly different than students. The results of 
the ANOVA on relationship with the principal are presented in Table 6. 

Table 6- ANOVA Results for relationship with the principal 







df 


F 


Sig. 


Relationship with the Principal Between Groups 


1 


3.439 


0.065* 




Within Groups 


303 








Total 


304 






Trust in Principal 


Between Groups 


1 


4.104 


0.044* 




Within Groups 


308 








Total 


309 






Respect for Principal 


Between Groups 


1 


0.418 


0.519 




Within Groups 


308 








Total 


309 






Cooperation w/ Principal 


Between Groups 


1 


6.114 


0.014* 




Within Groups 


307 








Total 


308 






Collaboration w/ principal 


Between Groups 


1 


4.993 


0.026* 




Within Groups 


306 








Total 


307 







* p<0.05 



As shown in Table 6, another significant finding of the study was the difference in relationship with 
principal between teachers and students. According to the results, all measures of relationship except “Respect for 
Principal” were significantly different between the groups. 

The results of the ANOVA on relationship with students’ readiness for assuming responsibility for their 
own learning are presented in Table 7. 

Table 7- ANOVA Results for students’ readiness for assuming responsibility for their own learning 





df 


F 


Sig. 


Between Groups 


1 


75.439 


0.000* 


Within Groups 


301 






Total 


302 







* p<0.05 



As shown in Table 6, ANOVA result points on significant difference on teachers and students perception on 
readiness of students assuming responsibility for their own learning. Although students perceive themselves ready to 
take responsibility teachers don’t see them as individuals responsible for their own learning. 

Conclusion 

Some of the conclusions that can be drawn from the study’s findings are as follows. According to the 
analysis on students’ readiness for taking responsibility of their own learning, teachers do not view students as ready 
for the change. This might be related to the view about the child. Teachers might have a tendency to view students 
closer to the industrial-age view. There is a possibility that students are perceived as passive learners and as 
everything the school has turned them into: passive, unmotivated, lacking skills for self-directed learning. This of 
course, is a speculation for now and needs further empirical studies to talk with more certainty. 
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Students’ relationship with the principal comes up to be weaker than teachers with the principal. This could 
be due to the organizational hierarchy. Simply stated, teachers interact with principals, and students interact with 
teachers. There seems to be no direct relationship between students and principal. These scores being low might 
also be from students viewing principals as the authority in charge of punishment. The industrial-age notion of 
sending students to principal for punishment might put the principal into a role that makes them naturally less 
friendly with the student. 
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Speak Friend and Enter: Using Asynchronous Voice Conferencing to Design 

Community for Online Learning 

James F. Doran, Jr. 



Abstract 

Asynchronous voice conferencing (AVC) is a new technology that delivers human voice to the online 
classroom and satisfies distance learners’ need for the convenience of asynchronous activity. This phenomenological 
study investigated the social impact of AVC technology in an online learning environment that emphasized 
collaborative learning methods on the development of learning community, identity and presence. A month-long 
simulated class was offered to online learners using AVC to complete collaborative learning tasks. This paper will 
describe how using AVC impacted the development of a sense of community. 

Introduction 

In the past decade there has been a growing acceptance of online courses in American colleges and 
universities. Almost two-thirds of all schools offering face-to-face courses now also offer online courses (Allen & 
Seaman, 2005, November). Online courses now show wide adoption, penetrating the core curricula of American 
higher education in all types of courses, programs, and disciplines and at all levels (Allen & Seaman, 2008). 
Currently, one in four American college students are taking at least one online course (Allen & Seaman, 2010). 
Ninety percent of these online programs use asynchronous Internet (Waits & Lewis, 2004) which primarily uses 
electronic text for communication among classroom participants. Yet it remains a peculiar irony that although we 
have mastered global voice communication for over 100 years, the Internet has until recently been silent, dominated 
by text-based and graphics-based systems for interpersonal and social communication such as email, chat rooms, 
instant messaging, and social networking (Walther & Parks, 2002; Williams, Caplan, & Xiong, 2007). 

As a communication medium, electronic text creates problems for instructional designers and educators who 
wish to build a sense of community and engage in collaborative learning. Electronic text creates a virtual classroom 
where the instructor and learners cannot see or hear the other people with whom they communicate (Palloff & Pratt, 
1999). Words on a computer screen can easily mask the true identity of participants (Turkle, 1995; Williams et ah, 
2007) and this anonymity can create a disembodied experience for learners (Coghlan, 2003). Without the benefits of 
paralinguistic cues such as gesture, emphasis, and intonation, silent electronic text can be difficult to express 
emotion and nuance without risking a higher level of misunderstanding in online discussions than in face-to-face or 
voice communications (Ice, Curtis, Phillips, & Wells, 2007; Kruger, Epley, Parker, & Ng, 2005). For the blind or 
sight impaired, the lack of an audio interface offers obvious disadvantages for accessibility (Edwards, 1989; 
Ratanasit & Moore, 2005; Schwartz, 2004). Text by itself is problematic for content areas that require developing 
and assessing specific oral and audio competencies such as learning to speak and listen to a target language 
(McIntosh, Braul, & Chao, 2003). 

In contrast to electronic text, the human voice has been demonstrated to be a communication medium rich 
with emotion, intention, social relationship and personal identity (Kruger et ah, 2005; Nass & Brave, 2005). Voice 
has been used for human communication and instruction in our evolutionary history much longer than the relatively 
recent inventions of writing, print and electronic text (Ong, 1982/2002). Introducing the human voice into the 
online classroom can personalize the sense of disembodiment that sometimes comes with a text-based asynchronous 
online classroom (Coghlan, 2003). Voice has been shown to be more effective than text in fostering trust and 
cooperation (Jensen, Farnham, Drucker, & Kollock, 2000), in creating durable relationships (Morris, Nadler, 
Kurtzberg, & Thompson, 2002; Neuwirth, Chandhok, Charney, Wojahn, & Kim, 1994) and in communicating 
expressive meaning and nuance which is especially relevant in equivocal situations such as giving collaborative 
feedback (Chalfonte, Fish, & Kraut, 1991; Ice et ah, 2007; Kraut, Galegher, Fish, & Chalfonte, 1992; Neuwirth et 
ah, 1994). Listening to the authentic voices of classroom participants should restore some of the missing cues from 
text-based communication, giving a more robust personal identity to the participants, and thus strengthening the 
development of a sense of community within the virtual classroom and reducing the sense of loneliness and 
isolation. A number of technological developments such as streaming audio and video for compression (Wilkinson, 
2004) and the wide deployment of high speed broadband Internet access (Horrigan, 2009) have now made restoring 
voice to the silent world of online learning an option for course designers and educators to consider. 
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Adding synchronous voice technology to online communication has been shown to have generally positive 
social effects (Card, Polin, Parra, Rhoads, & Sartori, 2006; Lightner, 2007; McCurdy & Schroeder, 2006; Rourke & 
Anderson, 2002; Williams et al., 2007). However, for distance learners it has the serious drawback of requiring the 
participants to meet at the same time. For many working adults in distance learning, the work day schedule and 
family responsibilities limit the time available for synchronous online learning, a restriction exacerbated by the fact 
that many distance learners are dispersed across multiple time zones (Lightner, 2007; Nibourg, 2005; Rourke & 
Anderson, 2002). Many of these learners were originally attracted to online learning by the prospect of “anytime, 
anywhere” education (Khan, 2001). A learner-centered approach to education must respect these essential needs for 
time flexibility that originally brought these learners to asynchronous education programs. 

Two asynchronous voice technologies can meet the need for time flexibility: podcasting and asynchronous 
voice conferencing. Podcasting allows digital audio or multimedia files to be posted and distributed to online 
subscribers who can play the files either on their computers or hand-held digital devices, such as iPods. However, it 
is essentially a one-to-many media distribution channel. It does not create an interactive environment that allows 
many-to-many communication as does electronic text applications. The receivers of the multimedia file can take 
only the passive mode of listening, with no easy way to respond and interact with the sender inside the same 
technology. Podcasting supports an objectivist epistemology and pedagogy that views learning as primarily a 
transmission of knowledge. At the same time, it creates a serious limitation for those who see learning in the 
constructivist and social perspective as a creative development of interactions between teacher and students in both 
directions, and among the students themselves. 

Asynchronous voice conferencing (AVC), on the other hand, is a new technology that delivers human voice to 
online classroom participants and satisfies distance learners’ need for time-shifting while supporting a collaborative 
learning strategy by creating a many-to-many environment that mirrors text-based course management systems 
(Charle Poza, 2005; Kim, 2005; Marriott & Hiscock, 2002). It also allows vocal cues to establish personal identity in 
learning environments (Kim, 2005; Marriott & Hiscock, 2002; Woods & Keeler, 2001) to encourage the social 
bonds that support a sense of learning community, engagement, and allows the expressive power of voice to 
communicate affective meaning where text alone often fails (Ice et al., 2007). 

Keith Ross (2003), a research engineer and college instructor, developed AVC technology in the late 1990s 
by combining streaming audio technology to the web interface of threaded text messages. Ross (2003) created the 
basics of asynchronous voice conferencing into a commercial product which he named “Wimba”. As in existing 
text-based message boards, messages have text titles with identifying information and are threaded in an outline 
format to indicate topic arrangement. Instead of a text message, however, each message plays the voice recording 
made by a user. From the user’s perspective, Wimba allows the time-shifting capability of established text-based 
course management systems where online learners can record vocal messages and listen to vocal messages sent to 
course participants at times convenient to them (Annis, Hensel, Lundstrom, & Jones, 2003). Since Ross’ start-up in 
1999, Wimba has been bought by the educational software maker Blackboard who has developed it to be compatible 
with WebCT, Blackboard, and some open-source systems. More recently, it has expanded Wimba to offer both 
synchronous and asynchronous voice technology for online education. 

Since the creation of Wimba, two other AVC products have appeared as beta products: YackPack and 
Vaestro. B. J. Fogg, at Stanford University, created YackPack with the goals of fostering vocal group 
communication that will build relationships and communication (Fogg, n.d.). Vaestro was founded in 2006 by Matt 
Ready as a beta voice community website open to users from around the world (Vaestro, 2006). Both YackPack and 
Vaestro allow users to store recorded voice messages intended for a pre-selected group of people such as a course 
room, and to listen to vocal messages left by other participants. 

AVC creates a many-to-many communication environment which allows group vocal discussion and 
communication that is essential to a collaborative learning environment. Online learners were able to conduct a 
debate (McIntosh et al., 2003) offering multiple perspectives on assigned topics (Charle Poza, 2005), and post and 
respond to vocal messages left by other learners (Marriott & Hiscock, 2002). Some learners reported that the AVC 
environment fostered more tolerance for differing opinions than face-to-face debates (McIntosh et al., 2003). 

It is not known how the introduction of AVC impacts the development of online learning community. The 
study of community and the interactions on which it depends have become a central framework in education 
research (Arbaugh, 2004; Duffy & Kirkley, 2004; Gunawardena, 2004). Developing a sense of community has 
become an important characteristic of learning effectiveness (Arbaugh, 2002; Barab & Duffy, 2000; Brower, 2003; 
Palloff & Pratt, 1999; Rovai, 2001, 2002b) and quality (Garrison & Kanuka, 2004; Rovai, Wighting, & Liu, 2005; 
Song, Singleton, Hill, & Koh, 2004; Thompson & MacDonald, 2005). Research shows that a sense of community 
can be developed in online learning courses with correct design and skillful facilitators (Brown, 2001; Rovai, 2002a; 
Rovai & Ponton, 2005; Thompson & MacDonald, 2005; Vonderwell, 2003) which is sometimes equal to those of 
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traditional face-to-face courses (Rovai, 2002b, 2002c). However, text-based computer-mediated communication 
(CMC) with its limited social cues (Cutler, 1995; Walther & Burgoon, 1992) makes it more difficult especially for 
novice online learners to interact with fellow learners and to build trusting online relationships in the limited time 
available within a course (Wilson, Ludwig-Hardman, Thornam, & Dunlap, 2004) . Although there are many 
potential factors driving community levels (Rovai, 2002a, 2002c), the medium of text-based computer-mediated 
communication affects directly or indirectly many of these factors. 

A few studies address the question of how asynchronous voice discussions might impact forming a sense of 
online classroom community. In blended classes, during the asynchronous online voice discussions and debates 
section, McIntosh, Braul & Chao (2003) observed that the learners showed great enthusiasm for the learning 
exercises, and a preference to interact with learners with whom they had already established a connection in the 
face-to-face class meetings. In completely online courses with distance learners, there are two studies of 
asynchronous voice with contradictory results. Ice and his colleagues (2007) found that hearing the voice of the 
instructor giving feedback increased feelings of involvement and learning community interactions for online 
learners. On the other hand, Woods and Keeler (2001) found that hearing the voice of the instructor giving general 
encouragement in various amounts made no difference in the sense of online community or satisfaction with the 
overall learning experience, but it did suppress the required online textual discussions. Both of these studies used the 
imbedded technology approach where voice files made by the instructor were attached to email or text postings, and 
voice was given only to the instructor thus creating a primitive podcast. 

For educators and instructional designers who value social learning methods, AVC with its many-to-many 
environment satisfies distance learners’ need for time-shifting while supporting a collaborative learning strategy and 
a discussion format (Charle Poza, 2005; Kim, 2005; Marriott & Hiscock, 2002). It allows vocal cues to establish 
personal trait identity in learning environments (Kim, 2005; Marriott & Hiscock, 2002; Woods & Keeler, 2001) and 
to encourage the social bonds that support a sense of learning community. It allows the expressive power of voice to 
communicate affective meaning where text alone often fails, especially in situations of collaborative feedback (Ice et 
ah, 2007). Much is still to be learned about the factors that make AVC effective for online learning and how it can 
support the development of online community for learners. 

Purpose 

The purpose of this study was to investigate the social impact of asynchronous voice conferencing (AVC) 
technology in an online learning environment that emphasizes collaborative learning methods on the development of 
learning community, identity and presence. This paper is limited to reporting on the development of a sense of 
learning community. 



Method 



Overview of Research Design 

This study proceeded in two stages. The first stage consisted of offering a month long simulated class to a 
small group of online learners using AVC technology in the form of Vaestro. The second stage began at the end of 
the simulated class, when the participants were interviewed to understand how they experienced using Vaestro for 
an online classroom and how they perceived their vocal interactions with their fellow learners. This study used the 
theoretical framework of phenomenology to elucidate the experiences of the participants. 

Sampling Design 

A criterion sampling technique was used to identify a purposeful sample (Miles & Huberman, 1984; Patton, 
2002) of XYZ University learners currently enrolled in the Instructional Design for Online Learning section of the 
School of Education. Ideally, a phenomenological study can seek out volunteers from those who already 
experienced the phenomenon under study. However, text-based communication is still the prevailing norm in online 
education (Card et al., 2006), so there was little chance of finding a large pool of experienced AVC learners. To 
overcome this limitation, a month-long simulated class was conducted using Vaestro as the AVC technology for 
XYZ University volunteers. These volunteers fit the criteria of being 18 years or older, had access to a working 
computer with a working sound card, stated that they were comfortable speaking conversational English, and agreed 
to learn how to use AVC technology for communicating within this virtual classroom. Participation in the study was 
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on a voluntary basis without any course credit and a token gift of $25 was offered to each participant at the end of 
the study in appreciation of the time and effort made. 

Participants 

Initial contacts were made by an individual recruitment email message sent by the researcher to 524 
learners enrolled in the Master and Doctorate programs in Instructional Design for Online Learning department at 
XYZ University. At the start of the online voice class, the sample contained 45 participants. In order to facilitate 
online discussions with smaller groups, the participants were randomly assigned into two sections of 22 and 23 
learners. A technical orientation and training was offered before the course began as recommended by previous 
studies (Charle Poza, 2005; Kim, 2005; McIntosh et al., 2003; Nibourg, 2005). Due to time pressures from other 
commitments, loss of employment, health issues, difficulties managing Vaestro, and sometimes for unknown 
reasons, 27 participants withdrew from the study, leaving 18 final participants for the interviews. 

The age of the 18 participants ranged from 29 to 63 with an average age of 49 years old. Females 
predominated in this sample as they did in the original list of 524 learners contacted by email. From section one 
there were 8 participants (2 males, 6 females) and from section two there were 10 participants (2 males, 8 females). 
The participants came from five different time zones. All of the online voice discussions were conducted in English 
and there was a broad range of various regional voice accents spoken by the participants. Three participants were in 
the Masters program and 15 participants were in the PhD program. All participants reported on the background 
questionnaire that they had completed four or more online courses in the past and none had prior experience using 
AVC technology. Participation in the month-long online voice class varied among the participants. Two participants 
dropped out after the second week, four participants did three of the four weeks, and twelve participants completed 
the assignments in all four weeks. 

Procedures 

The simulated online course lasted one month and used collaborative learning techniques to encourage 
online discussion (Palloff & Pratt, 1999). All course interactions were vocal using Vaestro. The initial task was for 
all the participants to introduce themselves to each other and to reply to at least two other participants. The 
subsequent assigned discussions involved a case study, a formal debate, a poetry recitation and the presentation of 
an “elevator speech.” The discussion was moderated by the researcher. Each week participants had one vocal 
assignment and were required to make two vocal postings offering peer feedback on their fellow learners’ postings. 

Data Collection and Analysis 

Data was gathered from three sources. First, background information about the participants was gathered 
from a short questionnaire completed by each participant at the start of the study. Secondly, the researcher collected 
impressions and observations from a one-month simulated online class using AVC. Lastly, after the completion of 
this online class, the researcher using a flexible question guide conducted two interviews over the telephone or 
Skype. Eighteen participants were then interviewed using a modified Seidman semi-structured process (Seidman, 
2006). Approximately one half of the interviews was transcribed by the researcher, and the other half was 
transcribed by a professional service, resulting in approximately 600 pages of typed transcript. All transcripts were 
mailed to the participants for their review to confirm the accuracy of the data. Their edits or additions were 
incorporated into the final collection of transcripts for analysis. 

This study used the four stages and procedures outlined by Moustakas (1994): epoche, phenomenological 
reduction, imaginative variation, and synthesis to analyze the collected data. The epoche, sometimes called 
bracketing, entailed a conscious effort by the researcher to set aside his experiences of asynchronous voice and to 
listen carefully to how participants expressed themselves, including their vocal emphasis on different aspects of their 
experience. This effort was maintained throughout the analysis of the data. Next, the reduction stage consisted of 
“horizontalizing” all statements (Moustakas, 1994, p. 90), giving them equal value. At this stage the relative 
importance of various statements kept shifting. It was necessary to re-read the 600-page transcript at least three 
times before patterns began to emerge. The imaginative variation stage consisted of an iterative process of creating 
categories that were winnowed, combined into larger categories or broken down into smaller categories. The 
researcher used tables in Microsoft Office™ and Inspiration ™ software to help in the analysis process, starting with 
607 categories and 17 tables. The final synthesis consisted of weaving the categories into eight major themes in 
order to answer the research questions. 
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Results 



An analysis of the data revealed eight major themes emerging from the interviews with the participants: 
recording, listening, interaction patterns, identity, group and community, presence, technical issues, and acceptance. 
This paper focuses on the fifth theme “group and community” and to a lesser extent the third theme “interaction 
patterns” to answer the question of how AVC affected the development of a learning community. 

Did a Sense of Community Develop? 

When asked if a sense of learning community developed, it became gradually clear that the participants had 
very different meanings of learning community that fell into three categories: relational, behavioral and structural. 
The most common response (Participants 2, 7, 8, 9, 10, 11, and 13) was that the four weeks of the AVC class was 
not enough time to develop connections with other learners they had never before met. These participants conceived 
of a learning community in a relational sense, that is, to be a collection of many interpersonal relationships with 
individual learners, not simply the structure of a class. Participants assumed that, if the voice class were to run for 
ten weeks comparable to the length of a XYZ University course, similar levels of community could develop. 

Well, in the [XYZ University] classes that are 10 weeks, sometimes I don't really start feeling connected 
until about [week] 7 or 8, just because of the pressure to post by mid-week and this discussion closes by 
Sunday, and then we are going to lock the thread afterwards. ... (Participant 8) 

Well, again, because it's shorter, it was similar .... If you had a course that was longer that used voice, I 
think that you would develop relationships in the same way.... they would take time; for me four weeks was 
not enough time to have the time to get invested or even want to because of distractions. (Participant 7) 

On the other hand, a few participants thought that a sense of a learning community did develop in our AVC 
class. Participant 1 8 thought we had built a learning community which allowed her to expand on her relationships 
that existed prior to the start of the study. She also recognized that hearing the voices of previously unknown 
learners could prompt an interaction. Two other participants (15 and 17) also thought that a learning community had 
developed. They both used the term, however, in the second category of a structural definition. An online classroom, 
even our simulated course, was a learning community by definition of being a classroom. 

In the third category of behavioral definition, posting learning assignments and replying to other learners 
itself was the community-in-action. Therefore, when individual learners made more posts and more learners 
frequently made posts that greater communication activity indicated stronger community. When the number and 
frequency of posts faltered or fell off, the learning community was perceived as waning. In this definition of 
community, it did not matter if the posts were text or voice; it was the level of posting activity that expressed either a 
strong or weak community. From this perspective, Participants 3 and 8 noted that the level of posting in the AVC 
course was often only the minimum required, perhaps due to the fact that it was a simulated course without any 
impact on grades and graduation. 

In summation, to answer the question of whether a sense of learning community had developed, most 
participants used a relational definition of learning community and felt that the four weeks of the online AVC class 
was not sufficient time to develop a sense of learning community. However, they assumed that, given more time, a 
sense of community could develop similar to the levels felt during a regular XYZ University online course with 
mainly text-based communication. 

What Was the Impact of Using AVC in Building Learning Community? 

Most participants found that using AVC made either a small difference or no difference at all in the 
development of online learning community, although for different reasons. Participants 1, 7, 9, 14, 17 did not see 
much difference between asynchronous text and asynchronous voice. Participants 2 and 4 thought AVC might make 
a small but not critical difference. Participants 2 and 12 found that all online media made human interactions and 
community building a very difficult task. Several participants thought that AVC made little impact on building 
learning community because other factors were more important such as time spent on the course (Participant 10), or 
the volume of posts (Participant 3). Participant 4 considered leadership to be more important that the question of 
voice or text: 
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It [voice] does make a difference, but it's not going to make or break the community one way or the other. 
For example, if it was a voice course room, and you didn't have a strong leader or a good community 
atmosphere, it wouldn't matter if it was voice. People would still just do whatever was mandatory or what 
they thought was necessary to pass the class. If you had text, it could be the same thing. You need that good 
leader and that collegiate atmosphere of academic trust, so that you can have discourse and you can have an 
actual community. Whether its text or voice, you need the community experience, you need a leader who is 
willing to say, "This is what is; this is how we can interact." And you have to have participants who are 
willing to voice opinions, positive and negative. If you don't have those from the learners and the instructor, 
you can't have a community, regardless of what medium you are using. (Participant 4) 

On the other hand, three participants (5, 13, and 18) found that talking and hearing others in the AVC 
environment had a positive influence on creating a learning community. Participant 13 thought that the added 
clarity that comes with voice in terms of inflections and emotions should help avoid misunderstandings, especially 
in the area of critical feedback given by peers, and that should contribute to the development of a learning 
community: 

One way I think that [AVC] can help, with respect to a learning community in general, is many of them are 
based on the notion of a peer review. And a lot of people struggle with giving people feedback, particularly 
negative feedback.... Sometimes people will give negative feedback, and based upon how they write, you'll 
think they're mad at you, like they think you've insulted their intelligence or something like that. While, if 
you had the opportunity to talk to them, say over a cup of coffee or something, you'd find out, "Oh, no, they 
don't really feel that strongly about it. They're just trying to help, too." So, again, the fact that there's more 
of...meta information that is carried in non-verbals, whether it's the tone of your language and things like 
that . . .just enhances the communication. (Participant 13) 

What Is An Academic Relationship? 

Participants considered relationships with classmates to be more “academic” than personal, more like 
professional relationships with work colleagues than personal friendships. Participant 1 described an academic 
relationship as one "where we only communicate for means of completing the assignments". Participant 1 1 
described an academic relationship as being similar to a professional relationship with more formal vocabulary. The 
emotion that she could hear in voice postings made it somewhat more personal than usual for academic discussions. 
An academic relationship between fellow learners usually is limited by the duration of the course that brings learners 
together. Often the end of a course is the end of relationships, so there is no expectation of future interactions that 
might strengthen developing bonds. This creates what participant 2 called the “goldfish effect”. 

You know how they say a goldfish memory only lasts for as long as it takes them to swim around the 
bowl?... That's how I feel sometimes about the social presence of my fellow learners. I'm a goldfish. Every 
time I see them again, I can't remember anything about them, except their name. (Participant 2) 

Participants 7, 14, and others expressed the same difficulty of remembering the names from the voice class during 
the interviews about two months after the end of the online voice class. It was noticeable that participants who had 
met other learners in person at colloquia from previous experiences had no trouble remembering their names during 
the interviews. 

Private/Public Channel 

Several participants noted how they often develop a more personal relationship with other learners by using a more 
private channel such as personal email, outside the public online class room environment. When asked if the voice 
environment made any difference for getting to know fellow learners, Participant 15 said: 

Well, that's difficult to answer only because we didn't have the opportunity to engage in dialogue outside of 
this particular study. And so, generally again, in classes or a colloquium you meet the individual actually 
during classes, when you have discussions online just at the very beginning and then you may decide to 
exchange telephone numbers or email addresses and quite possibly meet. So we weren't given an avenue, at 
least I wasn't, that would extend the relationship. (Participant 15) 

Exchanging email addresses would open a private channel between learners and create the opportunity for a more 
personal connection to develop. 

“We Are All In The Same Boat" Metaphor 



63 




Three participants (3, 1 1 and 17) during the course of the interviews used the metaphor “we are all in the 
same boat” to describe their connection with fellow participants. It is useful to explore this metaphor because it 
opens another perspective on how participants saw the group outside the framework of learning community. A 
group of boat passengers would not normally be called a community. A boat ride often entails people from different 
circumstances coming together for a short period of time to travel to a common destination. During the limited time 
together, fellow passengers share the same circumstances and may be forced to work together or at least to cooperate 
for common survival. In the same way fellow learners during the limited time of an academic course are brought 
together for a common goal, and perhaps some common interests, but for a limited period of time, and not 
necessarily because they share personal bonds with each other. Boat passengers do not need to become best friends; 
they only need to maintain a reasonable level of harmony to work together. The common situation creates some 
common needs and a sense of equality, but these feelings are tempered by the knowledge that at the end of the boat 
trip, all the passengers are free to go their separate ways perhaps never to see each other again. The group 
connections then are not necessarily very deep or personal, but require a certain level of mutual accommodations to 
facilitate attaining the common learning goals. 

Stealth Cohorts 

For the existing relationships that participants brought with them into the AVC class, stronger relationships 
were created or cemented by meeting other learners in person at the required colloquia held throughout the country. 
Many participants talked about the importance of the colloquia organized by XYZ University for making friends and 
meeting fellow learners who may have only been names until that point. In the course of the interviews it emerged 
that three of the participants (6, 12, and 14) had each separately met a number of fellow learners at various 
colloquia, and their discussions had developed into a small group that in effect became a "stealth cohort" of fellow 
learners and friends. The learners in these cohorts continued to communicate through synchronous audio-visual and 
voice communication (Skype, cell phones), to plan courses together in order to work as a team when a course 
required a group project, although no fellow members of these stealth cohorts happened to be participants in this 
study. 

Interaction as Homework 

To a lesser extent, the theme of interaction patterns also shed some light on the meaning that participants 
gave to community and how it develops. Participants 8 and 12 thought that the requirement of making certain 
feedback postings to other learners was an inadequate approach for creating true human interaction, and that adding 
or replacing voice did not address this central weakness. Participant 12 offered a cogent analysis of how required 
peer feedback was a limited strategy for developing a true sense of learning community. 

. . . once everybody gets into answering the discussion questions and taking care of their homework and 
doing everything else, they don't really pay attention to other people in the classroom, and they don't read 
everybody else: they go "OK, I got to reply to two students; whom am I going to pick today? Boom. Boom. 
Those two. They might not even read all the discussions or they may not listen to all the discussions and I 
think that hurts that community, because then it becomes — it almost becomes homework to interact. Well 
it is homework either online, either version, with the Vaestro or with the typical typing everything in; it's 
homework and it's not interaction. And I think that interaction is the one thing we haven't been able to 
capture and build a community with, with any courses online. (Participant 12) 

Posting Strategy and Patterns 

As experienced online learners, the participants had already devised different preferred strategies for when 
they would post their initial task response and when they would post feedback and respond to peer feedback. They 
reported employing these same strategies when they used AVC in the online voice class. The participants who 
posted early in the week on a consistent basis began to notice each other and form a sub-group: 

I always try to post early so I had a better relationship with those people that post early because we were 
there and we were talking and by the time it got to the end of the week I would listen to the rest of the 
posts, but I wouldn't necessarily talk to them because I already did my [feedback] posts.... (Participant 10) 
Those posting patterns could also become a variable to explain how relationships develop in the online AVC class. 
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To summarize, after four weeks using AVC, participants believed that the sense of learning community was 
developing at the same slow speed they had experienced using asynchronous text, if learning community means the 
development of personal relationships. If the AVC course proceeded to a normal ten-week course or longer, the 
participants expected a sense of community commensurate with a text-based course would develop. The substitution 
of AVC with its additional social cues over asynchronous text did not significantly eliminate the extra challenges the 
participants found in electronic asynchronous text when trying to develop a sense of community. 

Discussion 

The question of how AVC affected the development of learning community was complicated by the 
participants’ three different meanings assigned to it: structural, behavioral, and relational. In the first category of 
relational definitions, learning community implied the development of interpersonal relationships with other learners 
strong enough to survive the end of the course. It is this relational definition of learning community that is 
commonly used in the research literature. For example, Rovai explicitly uses that sense in his definition of 
“classroom community” (Rovai, 2002b) and “school community” (Rovai, Wighting, & Lucking, 2004). In that 
relational sense, participants said that four weeks of using AVC are not long enough to develop personal 
relationships with other learners they had just met, and that a sense of community for most participants was 
developing at the same slow speed as asynchronous text. Although Rourke and Anderson (2002) observed that 
asynchronous voice was slower than synchronous voice in developing a sense of community, the greater personal 
trait identity in voice (Nass & Brave, 2005) made it reasonable to expect at the beginning of this study that 
asynchronous voice would nonetheless be faster than asynchronous text in developing a sense of community. This 
however was not found, at least after four weeks. However, participants in this study reported that if the simulated 
AVC course proceeded to a normal ten-week course or longer, they expected a sense of community commensurate 
with a text-based course would develop. 

In the second category of behavioral definition, posting learning assignments and replying to other learners 
itself was the social activity that defined the community in action. Therefore, when individual learners made more 
posts and more learners frequently made posts that greater communication activity indicated stronger community. 
When the number and frequency of posts faltered or fell off, the learning community was perceived as waning. In 
this definition of community, it did not matter if the posts were text or voice; it was the level of posting activity that 
expressed either a strong or weak community. Thus, the question of how AVC impacts learning community 
becomes much more difficult to ask. 

In the third category of structural definition, if a group was called a learning community by an educational 
authority or organization, then it became a de-facto learning community. In this sense three participants considered 
the AVC course to be a learning community because it was a learning community by definition. When using this 
meaning of learning community it does not really make sense to talk about “levels of community”, since a group 
either is or is not a learning community. The question of how AVC impacts learning community breaks down. This 
structural perspective is closer to calling an online class as a common identity group instead of a common bond 
group, to use the distinction made by Prentice, Miller and Lightdale (1994). In a common identity group people feel 
attached to a common abstract issue, concern or cause such as art groups, sports teams, and church groups. In 
common bond groups people are attached to the group through personal relationships such as fraternities or 
university eating clubs (Ren, Kraut, & Kielser, 2007). The boat metaphor (“we are all in the same boat”) suggests 
that some participants saw their AVC class as a common identity group where the common purpose and goals of a 
course overshadows the importance of interpersonal bonds among the members of that course. 

Returning to the question of developing personal relationships, the participants found many obstacles. 

Some of these are endemic to the asynchronous environment for both text and voice. Late posters are often ignored 
by early posters and thus feel excluded from the group. The public nature of the AVC channel might not be as 
conducive to personal relationship development as private channels such as person email messages. Course 
interactions from peer feedback are too shallow to support relational development, and the time constraints that 
learning tasks impose are another obstacle. The discovery of the stealth cohorts amongst these online learners 
outside this study illustrates how the participants still preferred to meet fellow learners face to face whenever 
possible. It also suggests that substituting AVC with its additional social cues to replace asynchronous text does not 
significantly eliminate the extra challenges Rovai (2002a, 2003) found in electronic text for both facilitators and 
learners when they try to create and maintain a sense of community. 

Previous voice research found that voice has two important effects for social interaction. First it reveals 
individual personal trait identity (Nass & Brave, 2005). Secondly, it expresses affective states that give more precise 
meaning to verbal content (Kruger et al., 2005; Nass & Brave, 2005). Participants in this study did find both effects 
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in a general way. From a phenomenological perspective, these established voice effects had many significantly 
divergent meanings within the life-world of the participants. The additional nuance given by voice was broadly 
recognized but valued differently. It was helpful to some, and nice but not necessary to others, especially given their 
comfort level with text. 

Moreover, the voice research literature emphasizes the continuity of the voice effects across all formats 
such as recorded (asynchronous), natural (synchronous), or synthetic (either asynchronous or synchronous) (Nass & 
Brave, 2005, p. 45). In contrast, this study points towards the discontinuity of voice effects especially between 
asynchronous and synchronous environments, suggesting that both asynchronous voice and asynchronous text slow 
the development of a sense of community relative to synchronous media. In effect, asynchronous voice shares some 
important common elements with asynchronous text, which suggests that some temporal effects of asynchronicity 
and synchronicity cut across the media of text and voice. 

Why did the introduction of AVC not have the desired effects on the online learning environment? It is 
possible that previous research on voice did not consider the possibility that the power of voice operates differently 
in an asynchronous environment than in a synchronous environment. There was no apparent evidence in this study 
that asynchronous voice gave a special advantage for trust, or cooperation in the interactions of peer learners. In 
hindsight, the previous literature reviewed showing strong positive results of voice usually relied on synchronous 
voice and compared it with asynchronous text. For example, in the studies that show voice fosters trust and 
cooperation over text, as in the Jensen study (Jensen et ah, 2000), the participants spoke synchronously over a 
speakerphone. In the study of the online gaming community (Williams et ah, 2007) participants who used both voice 
and text in their group deliberations used a synchronous VoIP channel to speak in real time. In the negotiation study 
that showed voice was better than text in creating durable relationships which survive the friction of competitive 
negotiation, the participants used brief synchronous telephone conversations for the voice condition (Morris et al 
2002 ). 

The results of the present study call into question the extent to which the results of synchronous voice 
research can be generalized for asynchronous voice. Did these previous comparison studies measure the differences 
between the media of voice and text, or the temporal differences between asynchronicity and synchronicity? Can all 
these results still be applied to asynchronous voice environments? Could the results of these studies be duplicated if 
asynchr onous voice replaced synchronous voice? 



Implications 

The results of this study suggest that developing a sense of community remain a challenge for the 
asynchronous voice environment, as much as it does for asynchronous text environment. The asynchronous 
environment, whether text-based or voice -based, creates a somewhat diminished social reality that cannot be 
overcome by a simple substitution of asynchronous voice for asynchronous text. It can be supported by structures 
and norms such as social introductions and matching public farewells. At the program level, especially for distance 
learners, if there is any opportunity for online learners to meet their instructors and other learners in person, then that 
should be encouraged and supported, since this will facilitate all future communications. 

Limitations and Suggestions for Future Research 

The online AVC course in this study had several unique features that may limit generalizations. The 
participants found that four weeks of using AVC in a simulated course room was not long enough to develop a sense 
of community, but that given the same length of time of a quarter or semester length course, they expected that the 
equivalent sense of community should develop as it does for a text-based course. A longer study would have to be 
done to examine if this indeed happens or if something else emerges. Second, it was a simulated course without the 
pressures of traditional assessments, grades, and tuition. Lastly, the four different learning methods used in the 
online course (elevator speech, discussion, debate, and poetry recitation) are unlikely to be found in such a 
compressed order in a normal online course. It is possible that a longer study using a credit-bearing course and a 
traditional curriculum may have different results or produce new phenomena not in evidence in this study. Lastly, 
there is the sample. A small number of participants (n=18) were drawn from a single online university based in the 
United States of America, and all participants were already experienced online learners. The participants were 
working at the graduate or doctorate level in the School of Education. None of the participants had used AVC 
before. Although they denied that it was a completely novel experience, longer usage may have changed 
perceptions. They all experienced a particular installation of AVC (Vaestro) which is a beta product and will 
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probably undergo future technical developments. As more AVC studies are completed it will be easier to determine 
if samples from different populations have a significant impact on generalizability. 

Conclusions 

Participants in this study demonstrated that they could use AVC for instructional purposes in an online 
environment. AVC increased emotional nuance and personal trait identity, but this did not accelerate the 
development of learning community if you define that as relational development. This study suggests that the 
temporal effects of asynchronicity diminish social reality across both text and voice environments. Participants 
needed additional time and effort to individuate their fellow learners if they had not already met them in person and 
not all participants saw the value in that investment. The time flexibility that online distance learners especially need 
and which asynchronicity affords comes with the cost of a diminished social reality that not even recorded voice in 
AVC can overcome. 
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Principles of Effective Professional Development for School Library Leaders: 

The Survey Says.... 
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Abstract 

The purpose of this study was to examine the professional development activities of practicing school 
librarians. To be prepared to teach thinking skills, school librarians work to enhance their own skills and 
competences. School librarians seek professional growth through ongoing exposure to learning communities, 
webinars, presentations, traditional district wide meetings and membership in professional organizations. This paper 
will provide an analysis of survey responses and interviews from 72 school library media specialists regarding their 
professional development experiences. Elements for successful workshops and other professional development 
opportunities are discussed with recommendations for professional development from a variety of resources. 

Introduction 

In defining a strong library/media program one factor seems to contribute more to student success than any 
other and that is the presence of a highly trained - highly educated librarian. The impact librarians may have on 
student academic achievement is well documented (Neuman, 2011; Harvey, 2010; Lamb, 2011). Studies have 
proven that media specialists can have a positive impact on student achievement (Hopkins & Zweizeg 1999; Lance, 
Welborn, & Hamilton-Pennell, 1993; Loertscher, & Todd, 2003). Based upon these reports, both practicing 
librarians and policy makers have seen the need to integrate the school library position fully into the instructional 
practice. Therefore, efforts to improve and enhance the instructional skills, competencies and interactions of 
librarians, have become a target of educational reform (Darling-Hammond & Richardson, 2009; Desimone, 2011; 
Kuhlthau, Maniotes, & Caspari, 2007). 

Effective professional development for school librarians is critical to the process of ensuring quality 
instruction in supporting teachers and facilitating student learning (Peacock, 2001; Darling-Hammond & 
Richardson, (2009). Emerging technologies in addition to changing educational reforms and mandates demand that 
librarians pursue ongoing professional growth over the course of their careers. In response to these demands to 
fulfill the mission of professional growth, and as suggested by the American Association for School Librarians 
(AASL, 2009), professionals seek a range of face-to-face and online environments from which to gain knowledge 
from the greater learning community and each other in order to be better prepared in the school library profession. 
School librarians, like classroom teachers find multiple opportunities for pursuit of growth, and varied avenues to 
increase knowledge and skills, therefore it is critical to determine what an effective professional development 
activity requires. Using the results of an online survey, this study looks at a comparison of the positive and negative 
elements of professional development as reported by school librarians. 

Need for Study 

The value of professional development is widely accepted by educators in every field of study. The 
American Library Association (2009) specifically proposed that professional development opportunities are 
essential and valuable for school librarians. Opportunities for professional development are always expanding with 
the increasing availability of webinars, virtual conferences, as well as the multitude of traditional face-to-face 
conferences. In view of the ever expanding opportunities for professional development, attention to the validity and 
quality of the experience offered is needed (Wayne, Yoon, Zhu, Cronen, & Garet, 2008). 

Empirical research suggests there are core factors that may influence the impact of professional 
development school librarians. Researchers for this study were interested in what practicing school librarians would 
report as factors that make professional development effective. We posed the following questions for the study: (1) 
what are the various experiences, formats, and modes of delivery for professional development, and (2) how has 
professional development impacted the teaching role of the school librarian? To answer these questions researchers 
designed and conducted a web administered survey. 
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Methodology 



Based upon a review of the empirical research suggesting factors for effective professional development, a 
survey was designed to gather self-reported information on the professional development activities of school 
librarians. The survey was formatted and disseminated using a web-based survey instrument, Qualtrics. To 
emphasize reasons for the study, introductory information was sent electronically to the selected population along 
with the survey links. The survey featured twenty-two questions designed to reflect current empirical research in 
professional development for the school librarian. A Likert-type scale was utilized to obtain quantitative data, 
allowing for self-reporting with individuals addressing each question. Open-ended questions allowed respondents to 
provide more detailed and individualized information. Upon culmination of the survey, research software the 
Statistical Package for Social Science (SPSS), was employed, in conjunction with Qualtrics, to facilitate a detailed 
analysis of the reported data. 



Participants 

Participants were drawn from the population of practicing school librarians. Contact information used to 
disseminate the survey to participants was available through local, state, and national school librarian listservs and 
databases. The listservs included contact information for current students and alumni in a Master of Library Science 
program at a large southeastern university. In addition, the survey was sent to LM-Net and AECT division of 
School Media and Technology. Demographic information required was collected with the survey. There were 
seventy-three completed surveys in the database reflecting responses from librarians in rural, urban, or mixed 
schools. Librarians identified schools as 43% rural, 24 % urban, and 33% mixed school systems. School enrollment 
varied broadly from a low of 236 students to a high of 2700 students. Of those completing the survey, 20% were 
national board certified. Participant experiences included a range of practicing librarians from the inexperienced to 
the experienced, with 63% having served 5 years or less in their professional careers. Table 1 shows the range of 
experience of respondents. 



Table 1 



Professional Experience Range of Respondents 



# 


Answer 




Response 


% 


1 


1 to 5 years 




45 


63% 


2 


6 to 1 0 years 


m 


9 


13% 




1 1 or more 








3 






18 


25% 




years 










Total 




72 


100% 



Review of the Survey 

“Professional Development Experiences for the School Media Specialist”, an instrument designed by 
researchers, measured data as reflected by current empirical studies. Seven critical themes had emerged through a 
review of the empirical studies concerning school librarian professional development. Themes used for 
development of survey questions included: (a) formats or design for professional development, (b) job-embedded 
experiences, (c) environments supporting diverse preferences of learning, (d) opportunities for practice of skills and 
collaboration with colleagues, (e) systematic design and evaluation of workshop presentations, (f) impact on 
teaching and learning, (g) correlation with school wide missions and goals, and (h) engagement in professional 
groups. For purposes of this paper each of the themes will be discussed in terms of the current literature and survey 
findings. 
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Current Research in Professional Development and Survey Findings 



Professional Development Format 

The first theme, looking at forms or designs for professional development, was crafted to determine a 
librarian’s preference for workshop design. The traditional workshop format available to librarians today has 
broadened from individually or group scheduled workshops, 14-1 day staff development sessions, conferences, and 
instructional classes, to include broader communities of practice engaged in ongoing professional development often 
using Web 2.0 tools. For this survey, participants were specifically queried as to whether their experiences were 
provided through traditional school wide channels or through individual interactions with colleagues and/or 
administrators. According to Valenza (2010) & Warlick (2010), school librarians are increasingly taking advantage 
of twenty-first century formats such as webinars, wikis, and blogging, among other evolving formats. The responses 
of librarians in this survey population as to the type of format, whether scheduled workshops or on-going initiatives 
in the learning communities, was significant to researchers in understanding the choices school librarians are making 
for their own professional growth. The format reported by 69% of school librarian respondents to be most often 
provided, was through school-wide workshops, however, 33% indicated they had completed on-line modules 
specifically for the professional development of school librarians. These findings reflect a growing interest by 
librarians in taking advantage of non-traditional sources for their continued growth. 

While the format of presentation for the respondents in this survey reflected the more traditional method for 
professional development, 52 % of respondents did report that they had also taken part in efforts to grow 
professionally through personal interactions with colleagues and administrators. These opportunities for 
professional growth might include dialog with other educators during staff meetings and informal dialog and 
conversations in the media center, classrooms, or even hallways. 

Contrary to findings in the literature, (Valenza, 2010; Warlick, 2010) librarians in this sample do not appear 
to be taking full advantage of twenty-first century formats such as webinars, wikis, and blogging, among other 
evolving formats, and instead are continuing to utilize school-wide workshops, although as reported there is a clear 
trend towards emerging formats. For those who are planning professional development opportunities for school 
librarians, increased emphasis on expanded formats could serve to accelerate this trend. 

Job Embedded Feedback 

Professional development is considered to be more effective when seamlessly linked to instruction and 
curriculum needs, assessment or standards (Darling-Hammond & Richardson, 2009). When the learning is linked to 
the actual teaching/working reality, as in teachers being able to immediately use what is taught in their work, lasting 
results are more likely. Survey respondents confirmed current literature related to job-embedded activities. Several 
responses to the open-ended questions confirm the importance of job embedded activities within the design of 
professional development experiences. For example job embedded experiences are: 

. . .ones that are the most effective are those that I can return to my library and immediately put 
into practice. . . 

While one-shot skills presentations are less meaningful. For example: 

When I attend something that I cannot use in a short time after the workshop, its effectiveness 
diminishes. 

This aligns with research which suggests when professional development is focused on current instructional needs 
and permits direct, involved hands-on interaction librarians and teachers experience an enhanced sense of success 
and understanding (Garet, Porter, Desimone, Birman, & Yoon, 2011). 

Overall, practicing school librarians were able to positively link the instruction they received with the 
teaching in which they were involved. Further the professional development in which they were involved challenged 
librarians to tiy new things as reported by 86% of respondents, reinforcing the idea that their learning activities as 
professionals provided opportunities to assess their own teaching and make needed changes. 

According to analysis of the data, there was a clearly reported positive relationship between school 
librarians who experience job embedded professional development with opportunities to try new things in their work 
environment ( r = .383 p < . 001, one tailed ). Further, based on the correlation analysis there was a negative 
relationship between “trying new things” and receiving professional development that was not seamlessly linked to 
instruction and curriculum needs, assessment or standards. ( (r = -246, p < .002, one tailed). 
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Based on this correlation, school librarians who participated in “one-shot” workshops, which were not 
clearly linked to instruction, curriculum or assessment, were not as motivated to try new skills as those who reported 
receiving job-embedded professional development experiences. 

Further, librarians who are able to work with teachers and other librarians, engaged in their work and 
reflecting upon their own successes or failures realize greater professional learning (Darling-Hammond & 
Richardson, 2009). This relates to survey responses suggesting informal interaction with others as a format for 
professional development. One survey respondent supported Darling, et al. (2009) earlier research regarding the 
concept of reflecting on what works and what does not work in their instructions expressing: 

The most effective workshops are those that allow time for the participants to network and share 
ideas that have worked in their classroom or library. It is hard to miss the passion that a teacher or 
librarian shows when they are sharing something or an idea that was a success in their library or 
classroom. 

Evidence supports the literature (Thessin & Starr, 2011) suggesting that teachers who can share strategies in an 
interactive learning community directly linking the professional development to the challenges they are facing 
experience more effective learning experiences. 

Regarding the actual presence of job-embedded feedback in professional development for librarians in this 
survey, results demonstrated that 32% of librarians considered “absolutely” the experiences of their professional 
development to be job-embedded in their day-to-day professional duties. Fifty-one (51%) percent reported that their 
professional development was “occasionally” embedded in day-to-day duties. Twenty-one (21%) of those surveyed 
reported that they could immediately put into practice what they had learned in a professional development activity 
and 76% reported that they could also sometimes put the information to use immediately. These findings point to a 
clear need for increased attention to authentic professional development tied to job embedded activities. 

Environments Supporting Diverse Preferences of Learning 

The importance of involving teacher learners in the process of developing and designing learning 
opportunities cannot be overstated (Thessin & Starr, 2011). When teachers perceive they have little input in the 
educational environment and planning of their own learning, criticisms and disinterest are likely to occur, as 
reported by one disenchanted respondent in the current survey who shared: 

Most workshops are redundant in nature, with the resulting information taught conflicting with the 
rigidity of county mandated schedules and teaching to the test. They teach at these workshops that 
we should do a, b and c, but then it is impossible to implement because everyone on the grade 
level has to be teaching the same thing at the same time. Teaching to the individual child is 
preached but then made impossible to put into practice. 

Based on responses to open-ended questions from these survey results it can be reported that librarians know 
what their student needs are as well as their own needs. Input from the librarian is needed when integrating any 
school or faculty improvement initiatives. Responses indicated that to encourage “buy-in” by those whom program 
planners hope to serve, professional development planners include librarians in determining activities supporting the 
diverse and specific needs of their participants. 

Opportunities for Practice of Skills and Collaboration with Colleagues 

Building a common focus and a concerted effort to meet school needs depends upon strong collaborative 
relationships with colleagues (Flarada, 2002; Thessin & Starr, 2011). When teachers and administrators work 
together to address student needs by engaging in a continuous process of instructional improvement, teaching and 
learning are strengthened (Lamb, 2011; Thessin & Starr, 2011). It is most helpful when educators from the same 
interest group or focus are able to participate continuously as a professional interactive learning community 
(Desimone, 2011). 

Strongly supporting the literature in this respect, one respondent reported, “I went with a group of 
kindergarten teachers and we designed a module that we later used at school on celebrations around the world.” 

This response demonstrates how an experience with a professional learning community offered a rich, ongoing 
boost to the curriculum as a result of an opportunity to put skills into practice while collaborating with colleagues 
with a common focus. When teachers, as a group, are able to establish learning goals, purposes, and accountability 
though ongoing work as a unit, instructional processes are strengthened (Thessin & Starr, 2011). Formal and 
informal learning communities enhance the overall educational process incorporating the strengths of many building 
upon similar objectives. 
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Systematic Design and Evaluation of Presentations 

A major concern in designing teacher learning activities is whether or not teachers will be able to work 
successfully through the learning process achieving the objectives needed to develop enhanced skills for use in their 
school programs (King, 2000). Involving participants in the planning and design, linked to instruction and 
curriculum, will naturally entail more active communication and interaction on the part of the participant (Darling- 
Hammond & Richardson, 2009). Further, designers of professional development must link skills development to 
evaluation to be able to determine the impact of the process on the life of the school. Evaluation may be in the form 
of observation, peer appraisal or other assessment (Peacock, 2001) which looks at the results of staff development 
on school success. The most effective workshops according to a respondent “required work on the part of 

participants and are always a shot in the arm focused on the media specialist's role in reading, writing and the 

life of the school.” Integral to the process is to determine the teacher’s learning through a process of evaluation 
linked to the design of the program or activity. 

Results from this survey also indicated the importance of linking skills development to evaluation 
following professional development through observation, peer appraisal, tests or work samples. Evaluation by peer 
or administrator observation was reported at 75%. Closely following that was standardized testing results for 
evaluation by 66%. Action research projects were reported at only 14%. 

Impact on Teaching and Learning 

Paul Cobb (1994) proposed that “learning should be viewed as both a process of active individual 
construction and a process of enculturation into the . . . practices of wider society” (p. 13). Cobb (1991) implied that 
ideally, the continued training in which educators participate would be evident in their classrooms. It is important 
that librarians see an impact of their learning on their individual teaching in the classroom and thereby on the greater 
learning community (Desimone, 2011). Attention to curriculum and instructional habits is critical to the process and 
ultimate success of the training (Kuhlthau, et ah, 2007). Studies have demonstrated that learning activities which 
are focused mainly on teaching behaviors demonstrate smaller impact on student learning than programs whose 
content is more clearly focused on actual teaching as connected to curriculum or student learning styles (Kennedy, 
1998). 

Teachers in this survey were asked to report whether or not they felt their students’ achievement was 
impacted by professional development. A positive 38% of librarians surveyed reported that they felt that student 
achievement was impacted as a result of their training practices. Additionally 61% reported that “sometimes” 
professional development impacted student achievement. This supports current literature (Hopkins & Zweizeg 1999; 
Lance, Welborn, & Hamilton-Pennell, 1993; Loertscher, & Todd, 2003) that a qualified librarian has a significant 
impact on student teaching. The findings of this survey indicate that librarians, strengthened by proactive, willing 
professional development and collaborating with other instructional staff can have a significant impact on student 
achievement. 

School wide missions and goals 

Professional development that goes to the core of the school, reflecting missions and goals supported by 
school leadership is likely to have a more ingrained effect on the planning, teaching, and habits of librarians who 
participate (Darling-Hammond & Richardson, 2009). King (2000) suggested that for professional development 
activities to be considered highly rated, a focus upon critical integration of concepts reflecting school mission and 
vision were needed. To support librarians, an objective for advanced training may include gaining knowledge in 
how to support school wide missions and demonstrate expertise and understanding when working with the faculty 
and other stakeholders. Unfortunately in this survey only 29% of participants reported that professional 
development in their school reflected the school’s vision and goals. These findings indicate little evidence that 
librarians were exposed to or participated in dialogue discussing school mission or vision. Librarians in this study 
supported those in an earlier study (King, 2000), emphasizing that too much time was spent on the latest catchy 
phrase and not enough on what was pertinent to the teacher. Librarians and other educators want to focus on the 
issues that mold and impact their individual settings, not so much the national buzzwords of the day. One of the 
stronger statements supporting including school wide missions in designing professional development as shared by a 
respondent in this survey was 
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The most effective workshops I have ever attended have been ones that are relevant to what I do or to what my 
school's goals are. These workshops give research based information and allow time to modify the techniques 
or strategies to my particular setting. 

Results indicated that professional development integrated into school improvement plans and missions was valued 
by librarians seeking to enhance their skills. 

Engagement in Professional Groups 

It is common for librarians to seek professional support from the groups and organizations that they are 
members of or are familiar with. This is reasonable as those organizations seek to specifically identify topics of 
interest to their constituents. Professional support groups and organizations have long played a role in facilitating, 
endorsing and in some cases certifying, continuing professional development programs (Richard & Genoni, 2008). 
Further, the collaborative and collegial membership in professional organizations may provide librarians opportunity 
to impact on the types of training being shared whether in the form of conferences, short courses, seminars, training 
programs, mentoring, publications or grant opportunities. The librarians in this survey reported broad involvement in 
a number of organizations including: American Library Association (ALA), International Society for Technology in 
Education (ISTE), North Carolina Technology in Education Society (NCTIES), Society for Information Technology 
and Teacher Education (SITE), Treasure Mountain, Association for Education Communication Technology 
(AECT), NC Dept of Public Instruction (NCDPI), LearnNC, North Carolina Center for Advancement of Teaching 
(NCCAT) and National Boards of Professional Teaching (NBPT). One responded reported that “I attended NCCAT 
Connections through Johnston County Schools. This was a yearlong workshop that ILT l's and ILT 2's attended to 
help promote skills for classroom management, assessment, creative lesson plans and teaching students of poverty.” 
Another shared “NCTIES, it is not only fun, but I always learn something valuable to take back to my school” while 
NCSLMA provided the “most effective workshops such as Big 6 and how to convert to a flexible schedule” for 
another. According to the literature (Richard & Genoni, 2008), professional organizations have demonstrated strong 
support for continued growth of school librarians with specific librarian focused topics. The findings of this survey 
supported current literature in that respect. Table 2 shows the professional organizations most reported by 
respondents. 

Table 2 

Professional Group Engagement of Respondents 





Professional Organization 




Response 


% 


1 


American Association of School 




29 


50% 


Librarians (AASL) 




2 


American Library Association (ALA) 




16 


28% 




3 


International Society for Technology in 
Education (National Education 
Computing Conference) 


- 


7 


12% 


4 


NC School Library Media 




35 


60% 


Association(NCSLMA) 




5 


NC Technology in Education Society 




14 


24% 


(NCTIES) 
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6 ; 


Society for Information Technology and 




A 


7% 




Teacher Education (SITE) 


■ 






7 


Treasure Mountain 


u 


5 


9% 


0 


Association for Education 




Q 


1 WL 




Communication Technology (AECT) 








9 


NC Dept of Public Instruction (NCDPI) 




38 


66% 


10 


LearnNC 




20 


34% 



Elements for Successful Professional Development 

Implications of the analysis of data collected through the survey “Professional Development Experiences 
for the School Media Specialist point to a combination of strategies for those planning or seeking professional 
development. To ensure clarity and effectiveness and to design workshops that can be meaningful, attention to each 
of the areas examined within the survey is critical. The themes identified in this study suggest that librarians want 
opportunities for relevant training that can transform their practice. 

Multiple models and formats for continued learning exist and are continuing to evolve. Whatever format 
is used, relevancy of content is particularly important for effective professional development. Librarians also clearly 
want personal interaction with colleagues and ongoing initiatives that feature job-embedded activities. Most helpful 
are the systematically designed professional development models that consider school wide missions and vision. 

Conclusions and Recommendations 

School librarians deserve and need quality staff development not just to satisfy AASL’s standard for life- 
long-learning, but for the greater purpose of being able to directly impact upon the learning of countless students. 
There is little doubt that the success of students depends on the effectiveness and qualifications of teachers and 
school librarians. School librarians who have been well prepared and who have kept up with evolving mandates, 
formats, and technology are more likely to have a positive impact on student learning. If professional development 
is to have a positive, lasting impact, it must represent more than the latest buzzword or discussion forum. 

Librarians, who are called to promote and support education for all stakeholders, must lead the way in continuous 
learning efforts that support and transform student learning. 

This study revealed that librarians in this sample do not appear, currently, to be taking full advantage of 
twenty-first century formats such as webinars, wikis, and blogging, among other evolving formats, and instead are 
continuing to utilize more traditional school-wide workshops. However there is a clear trend towards increasing use 
of emerging formats. Planners of professional development opportunities for school librarians might find it helpful 
to more closely examine expanded formats and the involvement of school librarians in those formats. Further, as 
respondents reported increasing interests in professional growth through personal interactions with colleagues and 
administrators, future study of specific examples of the interactions with colleagues and administrators is an area in 
which future qualitative study may prove beneficial. 

The best way to ensure the quality of effective ongoing learning initiatives for educators is to provide 
professional development that is of high quality, is systematically designed, fully embedded in the teachings of 
librarians, and infused with interaction among colleagues. It is also critical that skill level or experience, grade 
level, and subject matter all be considered when planning programs. Optimum learning for teachers should include 
opportunities that vary with the curriculums, pedagogies, technologies and characteristics of the educators and their 
fields of study. Teacher learning with objectives to continuously improve student learning must first consider the 
needs of the individual teacher. It is, therefore, important to bring the teacher or librarian into the equation when 
planning learning activities in order to know how to meet specific needs or preferences and to determine what is 
relevant for the group. Support for the unique needs of the educator is vital to the success of the professional 
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development. Sharing in the initial planning for staff learning, enhances a teacher or librarians understanding, 
acceptance, and support of the process. 

Evaluation should be a part of the planning process allowing for feedback from those participating. 
Sustained involvement with the concepts or strategies covered in this article offer a foundation for maximum growth 
and development. Districts and professional organization that support librarians with rich, effective learning 
experiences enable librarians to support students with similarly enhanced educational experiences, thereby 
transforming both learning and lives. 
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Index Descriptors: Microblogging & Evaluation 



Introduction 

In 2006, the public was introduced to a new social networking platform bringing forth the first 
microblogging web-based communication tool known as Twitter (Ebner, Lienhardt, Rohs, & Meyer, 2010). This 
service has seen remarkable growth in popularity and pervasive use with approximately 200 million users to-date 
(Twitter, 2011). Twitter allows the public to follow media outlets publishing current events, popular personalities, 
and communicate with personal contacts, via the Internet, SMS texts, and mobile smart phone applications. 

The widespread adoption of microblogging amongst the public has facilitated a shift from this tool’s 
original use as a purely social networking tool. Some now see Twitter as a potential facilitator to be harnessed for 
those working towards meeting their educational needs and learning goals. An associate professor at University of 
Plymouth and the chair of the Plymouth e-Learning Conference, Steve Wheeler, has suggested several teaching 
strategies, in which microblogging can be used to redesign learning environments (Wheeler, 2009). 

These strategies were integrated into coursework designed for a preK-12 teacher preparation program and 
taught by three instructors at a large university in the southwester U.S. In place of the more prevalent asynchronous 
discussion boards, Twitter was used as a way to communicate with students enrolled in four different sections of a 
technology integration course. This particular microblogging tool was chosen for the online discourse due to its 
high level of availability. Students are able to access it across common personal computing and smart phone devices 
with access to the World Wide Web, as well as cell phones limited to SMS texting capabilities. 

Literature Review 

Microblogging, a Web 2.0 technology has been defined as a new form of blogging that allows individuals 
to post primarily text-based updates sometimes including links to images within 140-200 characters (Grosseck & 
Holotescu, 2009). The integration of social media such as tools like Twitter has been linked with engaging students 
in ways that move learning beyond the typical tools previously only found inside of typical LMSs (Mott, 2010). For 
example with the integration of public microblogging sites like Twitter, students now have access to a multitude of 
real life professionals who reflect on relevant professional experiences in numbers uncommon to previous 
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educational program curriculum design. These connections can then be leveraged towards maintaining students’ 
levels of engagement and interest in coursework, which typically have positive impacts on learning outcomes. 

Several studies have recently documented the use of Twitter as a beneficial pedagogical tool for 
maintaining student engagement. The use of Twitter was suggested to result in increased interaction between 
students, instructor, and the course material (Rinaldo, Tapp, & Laverie, 2011). In another study, Twitter was 
successfully used as an unrestricted communication tool to facilitate students’ informal learning (Ebner, Lienhardt, 
Rohs, & Meyer, 2010). Twitter has also been shown to result in positive attitudes when used as a tool to facilitate 
reflection on learning when faced with personal challenges (Wright, 2010). Junco, Heiberger, and Loken, 
documented significant differences in engagement and grade point averages between a treatment and experimental 
group of students using Twitter for their coursework (2010). Even though learning materials may need to be 
repurposed to avoid frequent interruptions when introducing the tool to students, Twitter was found to provide a 
benefit when connecting students with relevant real life scenario challenges found in classroom environments 
(Skiba, 2011). 

Preliminary studies have shown that when students are presented with learning through the integration of 
Twitter and course content, microblogging tools may in fact positively supplement traditional methods of instruction. 
When used effectively, it has been documented that Twitter can be a valuable tool to improve the quality of 
students’ learning experiences. However, the existing body of research about the potential outcomes when 
instructors might use Twitter to connect students across multiple course sections in attempt to sustain student 
engagement and supplement learning materials in a blended learning environment is limited. 

Program Overview 

The use of Twitter as a tool to support students’ learning was integrated as part of the course requirements 
across three sections of courses during the spring 2011 semester. As part of the initial instruction provided about the 
tool, students were first asked to create an account and follow their instructors and peers. Once they were orientated 
to the tool each student was prompted to reply to a discussion question using a distinctive hashtag (#). The 
instructors then showed students how to search for conversations using this hashtag. The instruction concluded with 
showing students how to search for people and common hashtags used to find information about areas of interest, 
which are associated with educational technology integration in K-12 learning environments. During each of the 
remaining weeks, instructors posted a link to an article or video and included a question prompt with a 
corresponding hashtag. Students were given a small amount of homework credit each week in return for responding 
to each prompt and at least one other peer’s Tweet. 

There were three key aspects associated with the tool’s integration. First, a web-based article or video was 
provided each week in support of a relevant topic, which was presented in the previous class or used to introduce a 
concept relevant to the following class. The instructors selected these resources and posted each of the links as a 
Tweet. Second, a corresponding discussion question was included such that each student was prompted to reply 
with his or her individual insight. Third, a distinctive hashtag was used so that students would be able to easily 
follow the asynchronous conversation and see other students’ comments. One hashtag was assigned to each separate 
topical prompt so that the group would be able to participate in each of the asynchronous discussions. This design 
allowed students enrolled in each of the course sections to seamlessly engage in the exchange of ideas and 
perspectives. As an example, one of the discussion prompts that was employed within this context is shown below: 

Watch [link to video]. What are some of the benefits that you could say to a teacher who says that 
digital storytelling is a waste of time or that it would be too hard for his students to do? Include 

#XYZ in your Tweet. 



Evaluation Method 

Participants 

The authors recruited 118 students, from a large university in the southwestern United States who were 
enrolled in a technology integration course for teachers with a preK-12 focus. This group was comprised of eighty- 
six undergraduate-level students with junior and senior standing (72.9%) as well as thirty-two students with 
graduate-level standing (27.1%). The mean age of this group was 23.8 years (SD = 5.32, range 19-47). The gender 
distribution was comprised of twenty-nine men (24.6%) and eighty-nine women (75.4%%). 
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Process 



The instructors introduced the microblogging technology to students during the second week of the 
semester. Each instructor taught his/her assigned section of students how to use the tool according to the procedures 
described above. The instructors awarded a small amount of homework points at the conclusion of each week’s 
asynchronous discussion. At the end of the semester, a questionnaire was used to capture students’ perspectives of 
the tool’s usefulness. 

Data Sources and Collection 

During the sixteen-week semester, students responded to roughly ten to twelve asynchronous discussions 
prompts. At the end of the course these students were provided with a questionnaire, designed to record their 
perceptions on the use of Twitter as a learning tool. The questionnaire included thirteen Likert items including a 
five-point scale (5=Strongly agree, 4=Somewhat Agree, 3=Neither Agree nor Disagree, 2=Somewhat Disagree, 
l=Strongly Disagree), presented below: 

1. I felt using Twitter throughout the semester contributed to building my professional teaching knowledge 
base. 

2. I feel that it was easy to learn to use Twitter for the purposes of this class. 

3. I enjoyed completing Twitter assignments in this class throughout the semester. 

4. I preferred to use Twitter over other forms of asynchronous discussions in this course (i.e. Discussion 
boards, Blogs). 

5. Using Twitter for class helped me learn about educational technology in the K-12 classroom. 

6. Twitter helped me feel connected with my classmates. 

7. Twitter helped me feel connected with my teacher. 

8. Twitter helped me feel connected with other K12 teachers outside of ASU. 

9. Twitter enhanced the development of the learning community in this course. 

10. Using Twitter positively effected my participation in asynchronous online discussions, as opposed to 
traditional discussion boards. 

11. I used Twitter to communicate with personal contacts before enrolling in this class. 

12. I used Twitter for educational and/or professional development before enrolling in this class. 

13. I will continue using Twitter for my educational and/or professional development. 

The following five open-ended questions were also posed to the student participants: 1) How much time did you 
spend using Twitter this semester? 2) How do you see Twitter being used as a learning tool? 3) What did you like 
most about using Twitter for this class this semester. 4) What did you like least about using Twitter for class this 
semester? 5) How would you improve the experience of using Twitter for class? 

Results 



Upon preliminary analysis of the quantitative data, the observed results revealed mixed student perceptions. 
As presented in Table 1, the students reported generally positive attitudes towards the integration of Twitter in the 
coursework. It also appears that students enjoyed the connectedness Twitter facilitated between their peers. 
However, it appears that the participants reacted negatively towards feelings of connectedness with their instructors. 
Lastly, it was interesting to note that most students had not previously used Twitter for personal or scholarly use. 

Table 1. 



Descriptive Statistics for Twitter Questionnaire 









Mean 


SD 


Attitudes 


1. 


ContrProfTeach 


3.160 


1.230 




2. 


EasytoLearn 


4.120 


1.260 




3. 


EnjoyAssign 


3.410 


1.470 




4. 


PreferTwit 


3.640 


1.570 




5. 


T witHelpEdT echinClass 


3.550 


1.330 


Connectedness 


6. 


TwitHelpConnectClassmates 


3.090 


1.310 




7. 


TwitHelpConnectTeacher 


3.750 


1.250 




8. 


T witHelpConnectOutsideASUT eacher 


2.640 


1.170 



80 




Participation 



9. TwitEnhnceLearninCommunity 


3.310 


1.280 


10. TwitAffectParticipation 


3.560 


1.420 


1 1 . UsedTwitBforePersonal 


1.800 


1.370 


12. UsedTwitBforeSch 


1.500 


1.010 


13. ContinueUseTwit 


2.910 


1.330 



N= 118 

A secondary analysis was then performed to further investigate potential differences resulting from the data 
collection between two distinctly different grade level groups of students; undergraduate and graduate students. A 
MANOVA between these two groups revealed that there were significant differences amongst students’ perceptions, 
dependent upon the participants’ grade level status. These differences are displayed in Table 2. 



Table 2. 



Twitter Questionnaire Analysis 

Undergrad Grad MANOVA 

(n=86) (n=32) Results 









Mean 


SD 


Mean 


SD 


F 


Sig. 


Attitudes 


1. 


ContrProfTeach 


2.980 


1.300 


3.660 


0.870 


7.472 


0.007 




2. 


EasytoLearn 


3.850 


1.340 


4.840 


0.570 


16.403 


0.000 




3. 


EnjoyAssign 


3.000 


1.410 


4.500 


0.980 


30.435 


0.000 




4. 


PreferTwit 


3.310 


1.620 


4.530 


0.980 


15.877 


0.000 




5. 


T witHelpEdT echinClass 


3.300 


1.410 


4.220 


0.750 


12.111 


0.001 


Connectedness 


6. 


TwitHelpConnectClassmates 


2.670 


1.170 


4.220 


0.940 


44.697 


0.000 




7. 


T witHelpConnectTeacher 


3.470 


1.300 


4.530 


0.670 


19.541 


0.000 




8. 


TwitHelpConnect 
Outside ASUT eacher 


2.570 


1.210 


2.840 


1.050 


1.275 


0.261 




9. 


TwitEnhnceLearninCommunity 


3.060 


1.330 


3.970 


0.860 


12.916 


0.000 


Participation 


10. 


T wit AffectParticipation 


3.260 


1.430 


4.380 


1.040 


16.302 


0.000 




11. 


UsedT witB forePersonal 


1.640 


1.250 


2.220 


1.620 


4.255 


0.041 




12. 


UsedTwitBforeSch 


1.420 


0.870 


1.720 


1.300 


2.076 


0.152 




13. 


ContinueUseTwit 


2.700 


1.350 


3.470 


1.110 


8.373 


0.005 



The analysis of the qualitative data was performed from a grounded theory approach (Strauss & Corbin, 
1994). The text-based responses from the open-ended questionnaire items were reviewed for recurring themes and 
descriptive codes were subsequently formulated. The results of this analysis revealed that students perceived 
Twitter as a tool, which can be used to support making connections between course concepts, discovering resources, 
and building community beyond the confines of a traditional classroom experience. 

When students were asked, “How much time did you spend using Twitter this semester?” responses varied 
widely. While a few students reported using Twitter for only “Five minutes each week,” some students described 
using Twitter for more extended periods of time with statements such as, “About two to three hours a week.” and 
“Five hours during the semester.” When the answers were further analyzed, three categories were developed to 
document the amount of time students seemed to spend using Twitter during the semester: 1) under one hour, 2) 
between one and two hours, and 3) over two hours. The majority of the undergraduate student responses described 
using Twitter under one hour during the semester. While the majority of graduate students reported using Twitter at 
least one to two hours during the semester. The graph presented in Figures 2 and 3 represents the frequency and 
distribution of responses between undergraduate and graduate students. 
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Twitter Usage by Time 
Spent 



Twitter Usage by Grade 
Level 




TimeSpent N - 118 



27 



22 



37 



Undergrad 



n Under 1 hour 
n 1 - 2 hours 
n Over 2 hours 



12 

14 

6 

Grad 



N- 118 



Figure 1. Frequency values representing the time 
students spent using Twitter. 



Figure 2. Frequency values representing the 
number of hours spent by each group. 



Students described several different ways they perceived using Twitter as a learning tool. For example, one 
student suggested, “Twitter polls could be a great way to get people to talk that might be reluctant to have 
conversation.” Another student suggested it was a, “Great way to check for understanding, encourage high level 
thinking outside of class, assist students in finding real-world applications for what we’re learning in class.” From 
the analysis of the responses gathered, there were four main themes that arose to describe students’ perceived uses of 
Twitter as a tool to support learning: 1) facilitate communication and discussion, 2) share and discover resources, 3) 
build a learning community, and as a 4) reflection tool. 

The results of asking students what they liked most about using Twitter in this context varied widely. 

Many students noted that they enjoyed discovering resources, the discussions, feeling connections with classmates 
and the professor, the efficiency and ease of use, concise communication requirements, learning to use a new tool, 
and the convenience of the mobile platform. A breakdown of the responses between undergraduate and graduate 
student input is presented below in Figure 4. A sample of the responses gathered is presented below: 

• 1 liked that what we had to write was concise, but I also needed to be meaningful. 

• Connecting with other educators and sharing ideas and links. 

• There wasn't any pressure to elaborate in your response, since answers were supposed to be short. 

• I got to see what my classmates thought about certain topics and there were always educational videos 

or websites to look through at my leisure. 

• It can be used as a classroom discussion board and connect the students to outside sources. 



82 



I could check in easily from my phone any time I wanted to and I could comment from my phone the 
moment a thought occurred to me. 



What Students Liked Most 





■Undergraduate 



Figure 3. Frequency values representing common themes found in most liked responses between student 
groups. 



Students were also asked what they liked least about using Twitter for class. The observed responses 
primarily focused on the complex organization and not being able to fully grasp the usability features of the tool. 
The undergraduate students also seemed to have more objections to the tool’s use than did the graduate students. A 
breakdown of the responses between undergraduate and graduate student input is presented below in Figure 4. A 
sample of the responses gathered is presented below: 

• Flard to keep track of when to post, where to post, or to remember to post. 

• I don’t feel like twitter is user friendly at all, it is difficult to search and at times there are so many posts it 
is difficult to locate and access the one we are supposed to be using. 

• That I had to get a twitter account. 

• Twitter isn't very convenient unless you have a smartphone, which I don't have. 

• There is very rarely enough space to write a coherent response. 



What Students Liked Least 




■^Graduate 
■ Undergraduate 



Figure 4. Frequency values representing common themes found in least liked responses between student 
groups. 



83 




The suggestions that students provided as to how would they would improve the experience of using 
Twitter for class varied according to their course section enrollment. The undergraduate students tended to suggest 
imposing additional structure in the workflow. For example, most of the suggestions made by the undergraduate 
students related to increasing the student-to-student communication requirements, providing reminders to post, and 
scheduling regular times for posting discussion prompts. Flowever, the suggestions made by the graduate students 
focused on providing additional instruction on the tool’s features including privacy controls, direct messaging, 
@mentions, and creating lists. 

Discussion 

Asynchronous discussion is often used as an instructional strategy within university-level blended 
coursework. In the past the most common tool used to support this type of content-based discussions has been a 
discussion board feature, which is native to many of the popular learning management systems available. It should 
be noted that each of the instructors were not previously experienced with using the tool and nor had they previously 
introduced it to their students. While some positive results were noted within the data collected as a result of this 
study, several modifications to the instructional process were derived from the students’ feedback for future 
integration in the classroom. The modifications to be made will include the following aspects: 1) a more focused 
and deeper introduction with the tool’s usability features, 2) additional direction geared towards discovering peers’ 
comments, and 3) additional instruction planned with outcomes designed for students’ to build a rational for and 
activate a personal learning network. 

Not surprisingly, we found that the graduate students who participated in this study reported spending more 
time outside of class using Twitter as a learning tool to supplement the course instruction. Since these students are 
presently teaching in authentic K-12 settings, they may be more impressed with the real impact that technology has 
on their current students and be more willing to explore using new technologies for potential adoption. Anecdotally, 
as college instructors we often note that students participating in graduate-level coursework seem to be more earnest 
and complete class assignments with more thought than do students enrolled in undergraduate-level courses. 

Conclusion 

During the past few years, researchers have documented microblogging and the use of Twitter to support 
student engagement inside of traditional lecture settings. We are now just beginning to acknowledge Twitter for its 
potential to expand the educational experience beyond traditional classroom settings. This case study covers one 
example of how Twitter was used by instructors to support extending student engagement with content materials and 
facilitating connections outside of face-to-face class settings. However, it is important to evaluate the use of Twitter 
in a large number of different contexts before any conclusions how the tool can best support student engagement and 
learning experiences. Adaptation of the methods presented in this case study for examination of the tool in different 
contexts would add valuable perspective and contribute to our collective knowledge about students’ perceptions of 
the tool’s use. 
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Abstract 

The purpose of this study was to explore the influence of the four traditionally hypothesized sources of self- 
efficacy on learners’ self-efficacy beliefs regarding learning mathematics in an asynchronous environment. The 
context of the study was a college algebra course offered in a mostly asynchronous, technology-intensive model at a 
public university in the southeastern United States. Participants (N=261) completed surveys assessing their self- 
efficacy to learn mathematics in this environment and to assess their mastery experiences, vicarious experiences, 
social persuasion, and affective/physiological state. Regression analysis indicated that in this learning environment, 
mastery experiences, persuasion, and affective/physiological state should be combined into a single dimension when 
considering the source of students’ self-efficacy beliefs toward mathematics success in the context of the study. 

Objective and Purposes 

Self-efficacy has been identified as an important construct for academic achievement in traditional learning 
environments for at least two decades. Its importance has been noted consistently through all levels of the 
educational process, with various student populations, and in varied domains of learning. Self-efficacy and its 
relationship to academic achievement in asynchronous online learning environments, however, are just beginning to 
be researched. Given the growing prominence of asynchronous online learning, it is essential that we understand 
what role constructs such as self-efficacy have in these learning environments. Albert Bandura’s (1977) 
introduction of self-efficacy theory included the proposition that self-efficacy is derived from four principal sources: 
mastery experiences, vicarious experience, social persuasion, and physiological/affective states. These four areas are 
generally accepted in the literature as the core elements in the development of self-efficacy beliefs, but an ordering 
of the importance of each of these sources is unsettled. Usher and Pajares (2006) summarize the inconsistent 
findings regarding the relative strength of each source well. They follow with the proposition that “exploring the 
predictive value of the sources of students’ academic self-efficacy beliefs and determining whether this prediction 
varies as a function of group membership such as gender, academic ability, and race/ethnicity is a matter of import.” 
(p. 130). 



86 




While learner self-efficacy has a well-established literature base in the context of traditional learning 
environments, self-efficacy research related to learners in online and other non-traditional learning environments is 
relatively new. Hodges (2008a) recently called for researchers to explore self-efficacy in online learning 
environments. The purpose of the present study is to investigate the relative strength of the four traditionally 
proposed sources of self-efficacy beliefs of students enrolled in a technology-intensive asynchronous college math 
course. 



Methods 

The research presented in this report began with data collection in January 2010 and concluded with data 
analysis, which was completed in April 2010. Participants in the study were solicited from all students enrolled in a 
college algebra course offered at a regional comprehensive, public university in the southeastern United States. The 
course was delivered using a technology intensive format. The students spent most of their time learning using 
instructional software and receiving just-in-time support in a math laboratory setting. Participants were invited to 
participate in this study during face-to-face meetings with the professors and also through email reminders. Out of 
448 students in the course, 271 students chose to participate in the study. A total of 261 students (133 female, 128 
male) completed all portions of the questionnaire. Their age ranged from 17 to 50, with an average of 19.74 years. 
Most of the participants listed their race/ethnicity as White (n=188), while Asian, Black, Hispanic, and Other were 
reported as well. The majority of students classified themselves as freshmen (n=178), while sophomore, junior and 
senior were also listed. Eighty-five percent of the participants in this study had never taken an asynchronous 
mathematics course. A large number of academic majors were reported in this study (n= 68). The three most 
common were undeclared (n — 34). business (n — 30), and the pre-professional field (n=27). Participants were not 
compensated in any way for their involvement with the study. 



The survey feature in Web CT Vista was used for data collection. Participants completed the Self-Efficacy 
for Learning Mathematics Asynchronously (SELMA) survey (Hodges, 2008b), a demographics survey, and the 
Sources of Mathematics Self-Efficacy (SMSE) scale (Lent, Lopez, & Bieschke, 1991). 

The SELMA survey is a 25-question survey constructed for use in college algebra and trigonometry 
courses offered in an asynchronous, technology-intensive format. In the present study it was found to have an 
internal consistency Cronbach alpha value of .9032 which is greater than the 0.80 level recommended as a minimum 
by Gable and Wolf (1993) for instruments in the affective domain and is consistent with other administrations of the 
survey (Hodges, 2008b). 

The SMSE scale consists of four, 10-question subscales designed to measure each of the four sources of 
self-efficacy: mastery, vicarious experiences, social persuasion, and affective/physiological state. The Cronbach 
alpha values observed for the SMSE subscales are listed in Table 1. 

Table 1 



Internal Consistency Values for SMSE subscales 



Subscale 


Mastery 


Vicarious 


Persuasion 


Affective/Physiological 


Cronbach Alpha 


.8974 


.6613 


.7971 


.9413 



The Cronbach alpha values observed for the SMSE in the present study are higher than, but comparable to, 
those reported by Lent, et al. (1991). Lent et al. observed internal consistencies of .86 for Mastery, .56 for 
vicarious, .74 for persuasion, and .90 for affective/physiological arousal. 
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Results 



The purpose of this study was to investigate the four traditional sources of self-efficacy beliefs of students 
in an asynchronous math course. A regression analysis was conducted. Each of the four subscales of the SMSE was 
used as a predictor variable for SELMA. Analysis of variance F (4,256)=27.31,/K. 0001 indicated a statistically 
significant relationship exists between the SMSE subscales and SELMA score. The results of the regression model 
are summarized in Table 2. 

Table 2 



Summary of SMSE Subscales predicting Self-Efficacy to Learn Mathematics Asynchronously (SELMA) 



Variable B 




SEB 




P 


Mastery 


.284 


.258 


.141 


.273 


Vicarious 


.313 


.187 


.097 


.096 


Persuasion 


.443 


.270 


.167 


.103 


Affective/Physiological 


.393 


.177 


.229 


.027 



Note, p <.05. r 2 =.299 



The results of the regression analysis indicate that of the four SMSE subscales, only the 
affective/physiological components were statistically significant predictors of self-efficacy to learn mathematics 
asynchronously. This result was unexpected since self-efficacy theory typically promotes these four areas as the 
main sources of self-efficacy. The lack of a statistically significant contribution by the mastery component was 
especially surprising, as mastery is typically posited to be the strongest component of self-efficacy belief 
development. 

To further examine these findings, the correlations between the predictor variables were examined. The 
correlations shown in Table 3 indicate a high degree of correlation between three of the variables: mastery, 
persuasion, and affective/physiological. The correlation between these sources is not unexpected. Usher and Pajares 
(2006) note that, “the sources informing self-efficacy are often intertwined” (p. 127). 

Table 3 



Correlations, Means, and Standard Deviations of SMSE Subscales 











Affective/ 


Subscale 


Mastery 


Vicarious 


Persuasion 


Physiological 


Mastery 


1 








Vicarious 


.345* 


1 






Persuasion 


.846* 


.428* 


1 




Affective/Physiological 


.860* 


.314* 


.748* 


1 


M 


31.9 


31.6 


32.5 


31.1 


SD 


8.7 


5.4 


6.6 


10.2 



Note. *p < .0001 



Multicollinearity is often suspected when predictor variables are highly correlated. Various rules of thumb 
exist for assessing multicollinearity, which involve examining tolerance or variance inflation factor (VIF). The 
tolerance and VIF for each predictor variable are provided in Table 4. Allison (1999, pp. 141-142) suggests that 
tolerances close to .40 and below should be flagged as possible indicators of multicollinearity. When Allison’s rule 





is applied to the values in Table 4, three of the four variables indicate possible multicollinearity issues. Field (2005, 
p. 196) summarizes additional literature regarding multicollinearity as follows: 

1. If the largest VIF is greater than 10 then there is cause for concern (Bowerman & O'Connell, 1990; Myers, 
1990). 

2. If the average VIF is substantially greater than 1 then the regression may be biased (Bowerman & 
O'Connell, 1990). 

3. Tolerance below .1 indicates a serious problem. 

4. Tolerance below .2 indicates a potential problem (Menard, 1995). 



Table 4 



Variable Inflation Factor ( VIF) and Tolerance for Predictor Variables 



Subscale 


Mastery 


Vicarious 


Persuasion 


Affective/Physiological 


VIF 


5.99 


1.23 


3.81 


3.87 


Tolerance 


.167 


.815 


.262 


.259 



If the rules collected by Fielding are applied to the data in Table 4, rules (1) and (3) indicate no potential 
problems in the present study due to multicollinearity, rule (4) indicates a potential problem with only one variable, 
and rule (2) may indicate a problem, if the average VIF of 3.7 is considered substantially greater than 1. O’Brien 
(2007) warns that rules of thumb for multicollinearity are sometimes applied in incorrect contexts, and that some 
techniques used to manage multicollinearity introduce more statistical problems than they solve. 

Since the literature regarding multicollinearity is in disagreement, and correlation of the predictor variables 
is known to be an issue in self-efficacy theory, multicollinearity will not be addressed in the present analysis with 
elaborate statistical methods. The unexpected results in the regression analysis (Table 2), combined with the high 
degree of correlation among predictor variables shown in Table 3 initiated further examination of the relationship 
between Self-Efficacy to Learn Mathematics Asynchronously (SELMA) and the predictor variables. Each predictor 
variable was regressed as a single predictor of SELMA and each was found to be a significant predictor as shown in 
Table 5. 



Table 5 

Summary’ of Regression Analyses for Individual Predictors of Self-Efficacy to Learn Mathematics Asynchronously 
(SELMA) ' 



Predictor Variable 


B 


SEB 


R 2 


Mastery 


1.03* 


0.107 


0.262 


Vicarious 


0.934* 


0.192 


0.08 


Persuasion 


$ 

1.31 


0.14 


0.249 


Affective/Physiological 


.0.868* 


0.09 


0.255 



Note, p < .0001 



Due to the high degree of correlation between the mastery, persuasion, and affective/physiological 
components, these three components were combined as a single measure denoted by MPA for mastery, persuasion, 
and affective/physiological. This grouping of intercorrelated predictor variables is suggested by Pedhazur and 
Schmelkin (1991, p. 451). When MPA and the vicarious components were used as predictor variables for SELMA, 
only MPA was found to be a significant predictor. Thus, the vicarious component was removed from the model and 
MPA was used as the sole predictor of SELMA. The results are shown in Table 6 and the reader should note that 
the R 2 in this final model is comparable to that found in the initial regression performed (Table 2). The elimination 
of the vicarious component is supported by its relatively low R 2 value as an individual predictor. 
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Table 6 

Summary of Regression Analyses MPA Predicting Self-Efficacy to Learn Mathematics As ynchronously (SELMA) 



Predictor Variable 


B 


SEB 


R 2 


MPA 

_ _ . 


0.393" 


0.038 


0.290 



Note, p < .0001 



Post-Hoc Analysis 

Several post-hoc analyses were conducted with demographic variables and SELMA. Only one statistically 
significant relationship was observed. Participants were asked whether or not they had taken Calculus in high 
school. An independent samples t-test was conducted to compare SELMA for those students indicating “yes” and 
those indicating “no” to that question. There was a significant difference in scores for those who indicated yes 
(n=53, M=122.17, SD= 16.6) and those who indicated “no” (n=208, M=1 12.46, SD=17.2); t (259)= -3.68, df =259, 
p = 0.0003. The effect size of this difference, 1 122. 1 7 - 1 12.46|/17.09 = 0.57, is categorized as medium effect size. 
This finding is not surprising since students who have previously had Calculus are likely to have had prior positive 
experiences with mathematics at the level of the course used in this study, thus enhancing their self-efficacy toward 
success through mastery experiences. 

Educational or scientific importance of the study 

Since the context of this study was a technology intensive, asynchronous learning environment, the results 
provide information related to recent calls (e.g. Hodges, 2008a) for investigations of self-efficacy in online learning 
environments. The present study also adds to the body of research on the sources of self-efficacy beliefs in varied 
academic domains called for by Usher and Pajares (2006). In addition, it provides supporting evidence to 
researchers who have hypothesized that self-efficacy beliefs are most impacted by mastery experiences. 

It is often stated in the self-efficacy research literature that the four primary sources of self-efficacy beliefs 
are the components examined in this study. The current study determined that only three of these four components 
were important in the context under consideration and that possibly, the three sources should be considered as one 
dimension. The model found (R 2 =0.29) leaves a significant amount of variance in self-efficacy to learn mathematics 
asynchronously unexplained. Prior research (Hodges & Murphy, 2009) has isolated the vicarious component, which 
was excluded in the current study, as the most important source of self-efficacy in similar contexts. Therefore, it 
appears that additional research regarding the sources of self-efficacy beliefs in these environments should be 
conducted. 

From a practical perspective, the results of this study are important to instructional designers and 
educational practitioners. The three sources identified in the present study are mastery experiences, social 
persuasion, and affective/physiological. Purposefully including elements to address these components of self- 
efficacy in an asynchronous course structured like the one highlighted in this study should enhance learner self- 
efficacy. Enhanced self-efficacy, in turn, should have a positive effect on achievement. Based on the results of this 
study, future research should be conducted to determine effective interventions and design guidelines to address the 
sources of self-efficacy identified here for learners in asynchronous online courses. 
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A Mixed Method Case Study of Student Engagement, Technology Use and 

High School Success 



Michele Jacobsen and Sharon Friesen, University of Calgary 
Jason Daniels and Stanley Varnhagen, University of Alberta 

Abstract: The relationship between technology use, student engagement and high school success was 
examined over two years. Surveys, interviews and classroom observations with students, teachers and 
school leaders informed study findings. Teachers are enthusiastic about technology, but design low level, 
low challenge recall assignments and tests. Student interest in learning with technology is high, but 
engagement tends to be low. Instruction is dominated by information delivery using display technology. 
Teachers must create meaningful, challenging and authentic student work that integrates technology. 
Leaders must create a shared vision for learning with technology in high schools; Research data on learning 
with technology must influence decision-making and systemic change in the education system. 

Overview 

To learn well in school, and to thrive and lead in a participatory and digital world, high school students 
need to be intellectually engaged in meaningful, challenging and complex work - work that is discipline rich, 
academically rigorous and motivates them to give over their hearts and minds to it. Intellectually engaging work 
motivates learners to challenge existing ideas, to build upon their passions and interests, and to develop 
explanations, arguments and solutions to problems that are complex enough to require collaborative teams of 
learners to investigate. It is important to move high school learning beyond the broadcast approaches that may (or 
may not) have served us well in the past 50 years. Today’s high school learners deserved to be engaged in 
participatory and technology enabled learning experiences and intellectually demanding inquiry projects in school 
(Bransford, Cocking & Brown, 2000; Jacobsen and Friesen, 2011; Jenkins, Clinton, Pumshotma, Robinson & 
Weigel, 2006). High school learners deserve to be well prepared for the ever-changing and digitally complex social, 
economic, political and cultural societies that they will inherit. The competencies and habits of mind that high 
school learners require to live and learn well in our media rich and socially connected global world differ from those 
even 10 years ago. As our participatory digital world keeps changing, school jurisdictions and high schools cannot 
afford to stand still. 

While research has demonstrated that knowledge building is a key requirement for learners in the 21 st 
century (Bransford, Cocking & Brown, 2000; Jacobsen, 2010a; Sawyer, 2006, 2008; Scardamalia, 2005; 

Scardamalia & Bereiter, 2003), most high school teaching is still characterized by information delivery and prepared 
messages for individuals to sit still and consume. High school students need to cultivate their ability to work 
collaboratively to improve ideas and share them publicly. School jurisdictions and high schools and need to deploy 
current participatory pedagogies based on current research in the learning sciences (Sawyer, 2006, 2008) and 
research on how people learn (Bransford, Cocking & Brown, 2000) to make collaborative knowledge building and 
intellectually engaged learning a reality for all learners and teachers. 

In the midst of a constantly changing and connected world, many high schools still do not or will not or 
cannot provide pervasive access to the robust technological infrastructures and network designs needed to serve 
citizens of a participatory and digital age well. While a small number of high schools now welcome student owned 
devices or provide 1-2-1 laptop access coupled with open, unfiltered networks for teaching and learning, most high 
schools struggle to maintain older computer labs and continue to dole out bookable timeslots; further, many school 
jurisdiction IT departments filter content and throttle the school networks, which limits further any innovative uses 
of technology. School jurisdictions need to put the proper technological resources in the hands of all learners and 
teachers. All stakeholders in education, from the Educational Ministry, to Universities, to the Professional 
Associations, to the school jurisdictions and community leaders, need to invest in and support teachers in designing 
intellectually engaging work for students to do - work that is worth their time and effort. 

The question that schools systems have to face is not whether this is the technology and media environment 
they want because this is the connected and digitally enhanced environment that we have - global, social and 
pervasive. Instead, high schools need to be asking how to change the way that teachers design learning experiences 
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for students and how leaders and the profession can better support teachers and students in making best use of 
modern technological resources and open connectivity. High schools need to become spaces in which learners with 
diverse strengths, interests, abilities and skills are brought together around collective interests to work 
collaboratively on shared goals and tasks, to create and share ideas, and to build and cultivate knowledge in a 
community. A challenge for high school is to reconcile impoverished technological infrastructures and locked down 
networks, and teacher-driven content delivery approaches with the collaborative knowledge building and 
participatory learning approaches and expectations of today’s high school students. Clearly, transformative changes 
are needed to move high schools from rhetoric to the realities of visible learning and visible teaching (Hattie, 2009) 
with technology in 21 st century learning contexts. 

Technology and High School Success 

Improving high school completion rates is a priority for the Government of Alberta and the provincial 
education ministry, Alberta Education. Recognizing that high school completion has both individual and societal 
benefits, Alberta Education works closely with school jurisdictions to explore and support innovative strategies to 
improve high school completion rates. It is well known that the effective use of technology can increase student 
engagement, impact student achievement, increase student and teacher ICT skills and, ultimately, change teaching 
practices. The Technology and High School Success (THSS) initiative was part of Alberta Education’s ongoing 
research into best practices in classroom technology implementation. In 2007, a Call for Proposals was issued to all 
publicly funded school jurisdictions and charter schools in Alberta for research-supported proposals that would 
explore the use of technology to improve student engagement and success in high school. The emphasis for this 
grant funding was on initiatives that demonstrated innovative uses of technology-enhanced learning environments to 
improve student learning and success in high school. In total, 24 school jurisdictions and/or charter schools were 
successful in receiving funding. A research team was funded to carry out the two-year, THSS research project in the 
2008/2009 and 2009/2010 school years. The THSS initiative involved over 22,000 students and 420 teachers at over 
70 schools in 24 school jurisdictions. The majority of classes involved in this initiative were Grades 9 to 12. In this 
paper, the authors present a selective overview of key findings from the two-year investigation. The research 
findings presented in this paper emphasize intellectual engagement, thoughtful and appropriate use of technology 
and the role of ongoing professional learning to support teacher development. 

Research Questions and Methodology 

The primary research question in the THSS study was, “What is the relationship between effective use of 
technologies, student engagement, and school success?” To answer this question, the research team explored a 
number of supporting questions: 

1 . What is the impact of technology on student engagement and success in school? 

2. How was technology used to support and enhance student learning? 

3. What are examples of successful models of professional learning and practice? 

4. What are the essential conditions, including technical, administrative, and facility considerations that 
are required to support classroom technology integration in secondary school environments? 

5. To what extent were the local goals of the projects achieved? 

6. To what extent were the provincial goals achieved? 

A mixed method case study approach was employed to answer the research question. Case study research 
intentionally focuses on the complexity of a single case, or a bounded system, as the phenomenon of interest for 
disciplined investigation (Merriam, 1997; Stake, 1995). A strength of case study research is the ability to examine, 
in-depth, a case or a system within its real-life context to describe what happened and why (Yin, 2009); conversely, 
case study is not an inferential method focused on describing causal relationships - hence, this study was not aimed 
at generalizable findings. The phenomenon of interest in this investigation is complex: student success in 23 of 24 
school jurisdictions that were participating in the Alberta Education funded Technology and High School Success 
Initiative (THSS) from 2008 - 2010. Thus, a range of research methods were called for in order to capture and 
describe the complexity of each case, and to facilitate cross-case synthesis and explanation building (Yin, 2009). 
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Both multiple methodology and mixed methods are educational research terms used to describe studies that 
include at least one qualitative and one quantitative research method to produce knowledge claims (Smith, 2006). 
Mixed methods research is an approach to research that is based upon the premise that research questions should 
dictate the methodologies used (Johnson & Onwuegbuzie, 2004). 

A mixed methods approach draws from the respective strengths of both qualitative and quantitative data 
collection and analysis methods and allows researchers flexibility in being able to mix and/or combine different 
approaches. The first reason for employing a mixed methods approach in this situation was for triangulation — 
leading to higher convergent validity through the use of multiple measures of similar underlying concepts — and the 
second reason was for complimentarity — examining different elements of a concept using different methods (Green, 
Caracelli, & Graham, 1989). The appropriate use of a mixed methods approach, employing certain research methods 
that fit with a range of research goals, was considered the most appropriate approach given the complexity of the 
research questions in this case study. 



Data Collection and Analysis 

Several data sources were used to document the impact and outcomes of the initiative in two years. Data 
collection methods ranged from online surveys of student engagement and technology use by teachers and students, 
to focus groups and interviews with students, teachers, school leaders and jurisdiction personnel, to collecting field 
notes from site visits and probing school and district records of school completion and student achievement, to 
conducting classroom observations using an established protocol. Provincial achievement data, student completions 
and attendance patterns were triangulated with researcher sources of data. 

Almost 50 schools in 23 different school jurisdictions were included; the research team focused on Grades 
9-12 students and teachers, and various jurisdictional personnel involved in the projects: district administrators, 
school-based administration, team leads and technical support advisors/teams. Data were collected from 
approximately 3400 participants at least once throughout the project’s duration (Table 1). In addition, two site visits 
were conducted in the 23 school jurisdictions over the two years of the project. Site visits provided opportunities for 
the research team to interview participants, conduct classroom observations and engage in ongoing dialogue with 
leaders and teachers involved in the Technology and High School Success initiative. 



Table 1: Total number of respondents 



Data collection method 


Year One 


Year Two 


Total 


Online survey 








Students 


1106 


1327 


2433 


Teachers 


128 


166 


294 


Interviews / Focus Groups 


52 


30 


82 


Classroom Observations (n=40) 
Teachers and Students 


400 


200 


600 


Estimated Total* 


1686 


1723 


3409 



*Any aggregate participant number in our study is, by necessity, an estimate. 



Online Survey 

Over 2600 students and teachers provided feedback and data for analysis via the online surveys during the 
two years of the study. The online student survey consisted of both select-response, fixed choice items and open- 
ended response items intended to gather more in-depth comments from respondents. Students were invited to 
participate in the survey through an initial email invitation and a follow-up, reminder email. The data collection 
occurred over a two-year period, with students completing the survey in both year one and again in year two. 
Students were not tracked from one year to the next. Two student survey instruments were used. One instrument 
was used to document students’ perceptions of intellectual engagement. Researchers got permission from the 
Canadian Education Association (CEA) to utilize parts of the survey developed for Willms, Friesen and Milton’s 
(2009) “What Did You Do In School Today?” study of secondary school student engagement. The other survey 
instmment measured students’ perceptions of technology. 
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Teachers were invited to complete an online survey about their instructional planning and practices. The 
URL for the online survey was sent to school authority contacts who then sent survey links to teachers in their 
school jurisdiction. The survey contained questions designed to gather information regarding teachers’ use of, 
comfort with, opinions of technology. The summary of survey data includes results from both the quantitative and 
qualitative data analysis. Quantitative data consisted of descriptive statistics. A content analysis was performed on 
the qualitative survey data, and participant comments were coded into various response categories. 

Interviews and Focus Groups 

Individual and focus group interviews were conducted throughout the two years of the study. A total of 82 
interviews took place over the two years (Table 1). Transcripts were initially read in their entirety for content and 
context, without imposing a specific analytic lens. In the second stage of transcript analysis, researchers read and 
coded each text independently to determine descriptive categories and criteria. Individual coding was then 
compared to establish consistency. These were a priori categories that arose from the research questions. Emerging 
themes that were outside the categories and criteria were noted and analyzed. The aim of this level of analysis was to 
both map out the data and review it for further analysis and to become more familiar with its content. In the final 
stage of the transcript data analysis, researchers reviewed critical criteria and descriptors, which created the criteria 
for content analysis for ongoing transcription analysis. 

Using the descriptive categories and criteria that emerged from the initial data analysis, the codes of interest 
were created, which formed the basis for the analysis. Once all the categories and criteria from the transcripts were 
determined, field notes were analyzed adding any additional categories or criteria. Then the research literature was 
consulted to further inform and validate the categories and criteria selected. In addition, the combination of codes of 
interest, research literature, and field notes were used to create the criteria for the content analysis. 

Researchers also analyzed the transcripts to discern patterns of experience. The transcripts were coded, 
noting all data that related to the patterns. The identified patterns were then expounded on and combined. Themes 
were defined, which were derived from patterns such as conversation topics, recurring vocabulary, recurring 
activities, meanings, and/or feelings. Themes that emerged from the participants’ accounts form a comprehensive 
picture of the collective experience. In this way, the researchers were able to establish which themes and sub-themes 
fit together in a meaningful way. 

Observational Analysis 

Classroom observations were conducted during both years of the study. Observations were made in 
classrooms using an established classroom observation protocol (Jacobsen, Saar and Friesen, 2010). Researchers 
asked to observe in classrooms in which the teacher was directly involved in the THSS project. Researchers 
conducted disciplined observations in classrooms, and during lessons, which were chosen / identified by the school, 
principal or classroom teacher. In addition, field notes were made throughout the two years of the study. Classroom 
interactions, student engagement and instructional practices were coded using three equal intervals during a lesson 
(beginning, middle and final third of classroom time). These observations were then aggregated and analyzed using 
a combination of descriptive statistics and qualitative content analysis. 

Key Relationships Between Student Engagement, Effective Use of Technology and Student Success 
Student Engagement 

Building upon Csikszentmihalyi's (1990, 1997) research on how people learn best - which is by doing 
things that are challenging and of deep interest to them, reflective of the close interplay of the emotional in cognition 
and the development of capacity - Friesen (2007) has defined intellectual engagement, the state in which the learner 
is so focused, so intensely engaged, that time itself seems to disappear, as a key goal for quality teaching and 
learning. An OECD report (2007) explains that at this point of engagement, the brain begins to make connections 
and see patterns in the information, which results in a “powerful illumination which comes from understanding” (p. 
72). This state of sudden epiphany is described as “the most intense pleasure the brain can experience in a learning 
context” (ibid., p. 73) and naturally, is an experience that fosters motivation as students experience the pleasure 
inherent in deep learning. 
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A number of researchers (Jacobsen and Friesen, 2011; Jacobsen, Saar and Friesen, 2010; Kuh, 2001, 2003; 
Means, Toyama, Murphy, Bakia, and Jones, 2009; Willms, Friesen & Milton, 2009) have focused on the 
connections among student engagement, the learning environment and teaching practices. These studies have 
shown that student engagement is related to a number of factors such as: (i) the types of instructional practices 
teachers enact, (ii) authenticity and complexity of the work students are asked to do (iii) the types of technologies 
students utilize in their learning and (iv) the amount and type of ongoing feedback students receive while they are 
learning. This research has established clear correlations between these school related factors present in the learning 
environment and students’ levels of engagement. Their research confirms a finding by the Learning Sciences and 
Brain Research project sponsored by OECD (2002, 2007). “The more closely the goals of teachers, learners and 
educational systems are matched, the more effective the learning will be. . . the more closely this learning is linked to 
external stimuli of ‘real world environment’, the more it will engage and stimulate the learner” (OECD, 2007, p. 
200). Based on rigorous research on engaged learning and engaged teaching, Friesen (2009) has developed a 
Teaching Effectiveness Framework to guide high quality design and support of student inquiry learning. 

Most of the 23 school jurisdictions involved in the Technology and High School Success initiative 
indicated that increased student engagement was a goal. Data from several sources, including interviews, survey and 
classroom observations, were triangulated to determine the extent to which students were engaged in their studies. 

On the survey, students reported on levels of their engagement in social studies, language arts, mathematics and 
science classes. During interviews and focus groups, students were asked to comment on the types and nature of 
tasks they were asked to complete across the curriculum, and the ways that they used technology to connect, 
communicate and collaborate. Classroom observation data was collected to explore various factors contributing to 
student engagement (i.e.. Instructional Practices, Authenticity and Complexity of Student Work, Assessment For 
and As Learning, and Academic and Intellectual Engagement). Levels of student engagement in lessons, tasks and 
activities was gauged at the beginning, middle and end of a lesson by counting the number of students displaying 
one of four types of behavior: 

• Disengagement would include inattention, attending to an alternative activity, off-topic conversation, or 
misbehaviour. 

• Ritualistic Compliance is identified as working on assigned activities without enthusiasm or personal 
investment. Going through the motions of completing work to avoid conflict or unpleasant consequences. 

• Academic engagement is identified by on-task behaviours that signal a serious engagement in class work; 
these include attentiveness, doing the assigned work, and showing enthusiasm for this work by taking 
initiative to raise questions, contribute to group activities and help peers. 

• Intellectual Engagement refers to an absorbing, creatively energizing focus requiring contemplation, 
interpretation, understanding, meaning-making and critique which results in a deep, personal commitment 
to explore and investigate an idea, issue, problem or question for a sustained period of time. 

From the perspective of student engagement, the first finding is that the majority of teachers participating in 
this study are in the early phases of adopting learner-centered instructional strategies; a teacher centered approach to 
lesson delivery in high school is not strongly correlated with student engagement. The second finding is that teacher 
activity and student groupings / interaction patterns indicate that the majority of classroom time is devoted to 
teacher-directed, whole group instruction rather than the student-directed, interactive, peer-to-peer interaction. The 
third finding is that the majority of participating secondary teachers are in the beginning phases of designing 
authentically engaging, complex tasks for students - for most high school students, the work they are asked to do is 
note-taking, answering pre-defined questions and completing chapter and unit tests. The fourth finding from direct 
classroom observations is that only one-third (30%) of teachers achieved an above average score (i.e., a score of 3 or 
4) on each of three measures of intellectual investment, instructional style and authenticity during 2-3 time 
intervals in a lesson. 

The most important thing a teacher can do to increase student engagement is to design and support student 
learning tasks that are meaningful, authentic and challenging (Jacobsen, 2010b). The classroom observation measure 
of task / activity authenticity indicates that the majority of tasks fall in the artificial versus the real world category - 
students are often asked to do replication work rather than knowledge building work in each discipline. Our fifth 
finding is that the majority of participating secondary teachers are in the beginning phases of designing and 
supporting learning environments that require and support intellectual investment by high school students. The sixth 
finding is that the majority of participating secondary teachers are well practiced at whole class instruction and 
guided, whole class discussion. The seventh key finding is that the majority of participating secondary teachers are 
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in the beginning phases of involving students in assessment and using constructive, timely feedback to improve 
learning. While we did not test the relationship with standardized achievement testing, there appears to be an over- 
emphasis on good marks in high school teaching rather than engaged learning and developing deep understanding. 

Finally, our eighth finding with regard to student engagement is that more than 50% of high school students 
exhibit disengagement and ritualistic compliance behaviors during the first third of class time in over 50% of the 
classrooms we visited. Disengagement and ritualistic compliance behaviors persist into the middle and final thirds of 
a lesson. Academic engagement was observed in less than 50% of the classrooms we visited, and the percentage of 
students who were academically engaged dropped as the lesson proceeded. Intellectual engagement was observed in 
very few classrooms in this study. In the six out of fifty classrooms where intellectual engagement was observed, it 
was the teacher’s connection to the discipline of study and the design of tasks that enabled students to work together 
on meaningful and challenging work, along with the appropriate use of technology for collaboration, expression and 
communication of ideas, and continuous assessment for learning, that were the conditions that led to the greatest 
amount of change and transformation 

Effective Use of Technology To Support and Enhance Student Learning 

Though there is a range of information, communication, social and participatory technologies that could be 
used in the classroom, in the current study, students reported that their teachers tended to use technologies such as 
interactive whiteboards and videos most frequently. Social and participatory technologies could, potentially, act as 
social levelers if used more frequently in school - for example, technologies offer students with mobility, hearing 
and visual challenges a “hand up” while many of these technologies could also be used to help all students to learn 
better in universally designed learning environments. While most teens are engaged frequently in social networking, 
the majority of schools and districts tend not to allow students to use social networking in schools. Students report 
that outside of school they frequently use social networking software (see Table 2). 

Table 2: Te chnology Frequently Used by Students Outside of School (Open-ended survey item) 



Comment category 


Number of comments 


Social networking tools (e.g., Facebook) 


286 


General internet use 


37 


Gaming 


21 


Cell phones and texting 


15 


Music 


13 



A majority of students (76%) revealed that when technology was used in the classroom, most often they 
were watching or listening to the teacher present material to the class while using technology or that they (70%) 
were working alone with technology. There is an observed disconnect between the technologies that students use 
outside of the classroom (Table 2), and are comfortable using, and the technologies that they are exposed to in the 
classroom, such as teacher-controlled interactive whiteboards, multimedia content and streaming video 
presentations. 

An analysis of the interviews with school and district staff suggest schools and districts are in the early 
stages of providing opportunities for students to make full use of a range of technologies for learning, particularly 
the technologies that sponsor deeper learning. The analysis of the interviews is supported by the survey results and 
the classroom observations. Given the widespread need for quality teacher professional development and continuous 
learning opportunities, coupled with uneven distribution and availability of robust technological infrastructures and 
networks, many school jurisdictions are not in the position to require that students in secondary schools to be 
expected to use of a wide range of technologies to develop information, media and technology skills. Given that the 
technological infrastructure and availability does not yet support ubiquitous and ready access to technology by 
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students, systemic solutions to getting the training to teachers and the right technologies in the hands of teachers and 
students must be sought. 

Cultivating global citizenship is a key priority to help prepare students for life beyond school. One benefit 
of technology is it can be used to overcome communication barriers. Student can gain access to professionals and 
other experts beyond their local communities, and connect with other students across the globe. On the survey, 
students were asked to indicate how often they used technology to collaborate or communicate with others. Overall, 
students felt that they did not have many opportunities to collaborate via instructional technology. Almost half 
(47%) indicated they worked frequently with other students in their classes, but collaboration with students outside 
of their classroom or their school environment was limited. Students are not using the communication potential of 
the technology that they have within their classrooms. Though students in the student interviews indicated that they 
frequently used cell phones and Facebook™ to communicate with friends, these technologies are not used in the 
classroom. From year one to year two, the reported weekly usage of computers to work with other students in the 
class increased from 12.6% to 16.1%. The reported weekly usage of computers to work with other students at the 
school increased from 9.2% to 12.8%. In both of these cases the reported daily usage increased as well, but just a 
little, which may indicate that the increase was because of students increasing” computer use from monthly or never 
to weekly. It is clear that with regard to appropriate, meaningful and challenging use of technology for intellectual 
engagement in high school, there is a great deal of room for improvement in many school jurisdictions. 

Discussion and Recommendations 

As indicated, this paper presents a selective and summarized overview of key findings from an intensive, 
two-year multiple case study in 23 school jurisdictions in Alberta. A 125 page research report was presented to the 
Alberta Education Ministry, and will become publicly available in 2012. Overall, researchers documented a few 
exciting innovations taking place in a small number of high schools. The many teachers who are supporting 
academically and intellectually engaging work for students are commended for the good work that they are 
designing for and with students, and for persevering with innovative practices, often in spite of system inertia. 
Differing levels of technological integration did occur in the various jurisdictions involved in the TFISS project, with 
more ritualistic use of technology in many divisions; the teacher is the main user of the technology and often 
employs technology using conventional, information delivery approaches with content acquisition testing methods. 
Given the paucity of meaningful and quality professional learning experiences that teachers are engaged in, and the 
impoverished technological infrastructures and locked down networks in many high schools, one can empathize 
with teachers who clearly want to innovate and do more intellectually engaging work with their students and who 
find themselves unable to actualize their visions and creative ideas with students. Teacher enthusiasm is not the 
problem; many teachers have a huge appetite for transformational practices and technology infused projects. While a 
small number of teachers are using technology in innovative, intellectually engaging ways, the majority are not; it is 
clear that some teachers are simply unable to use technology in the ways that are called for in the 21 st century given 
the large number of barriers (i.e., filtered and locked down networks, insufficient access to technology, little access 
for student use of technology, inadequate professional learning). Many teachers report that students are now in 
charge of their own learning and seem more engaged in the process as a result. Flowever, the benefits of technology- 
enabled learning for students and teachers were clearly not achieved by the majority of schools and school 
jurisdictions involved in this study. There is a gap between what teachers believe is intellectually engaging work, 
and what students experience as engaging work. 

The majority of participating secondary teachers are well practiced at whole class instruction and guided, 
whole class discussion and they are the first to admit that they want to make changes to their teaching. The majority 
of participating teachers are in the beginning phases of involving students in assessment and using constructive, 
timely feedback to improve learning - there is a clear role here for targeted professional development and learning. 
With regard to student engagement, more than 50% of high school students exhibit disengagement and ritualistic 
compliance behaviors during the first third of class time in over 50% of the classrooms we visited. Technology is 
not being deployed in ways that take advantage of students’ skilled and regular use of social and entertainment 
technologies. Students report the predominant use of technology within their classrooms is watching or listening to 
the teacher present material to the entire class. It is fairly evident that the technology is not in the hands of high 
school students in many classrooms; it is just as clear that technology should be in the hands of students. 
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In just less than 1/4 of the classrooms, students were provided with the opportunity to build 21st century 
skills such as scientific literacy, social or personal responsibility, the ability to use technology in real world ways, 
and self-direction. More opportunities must be intentionally designed into secondary learning experiences for the 
other 75% of students to develop 21st century competencies. In order to understand how to effectively design 
learning that uses technology to increase student engagement, teachers need high quality professional learning 
opportunities to engage and learn in similar ways themselves, and they need to subject their learning to peer review 
and critique (Jacobsen 2010b). 

The professional development provided to teachers tended to involve mostly technology training rather 
than pedagogical design. While some technology training is necessary to help teachers to learn to use technology, it 
is insufficient to allow teachers to effectively use technology in the classroom. The majority of teachers recognized 
that incorporating technology into learning requires them to redesign their instructional practices. When asked what 
their ideal vision of technology use would be, teachers most frequently cited increased access to technology and to 
professional learning with technology in their subject area. In the teacher survey, and in both the teachers focus 
groups and the student focus groups, a common concern from both teachers and students was lack of access to the 
internet because of internet filters and network firewalls. 

A key finding from this research is that although technology is increasingly being used in high school, it is 
still not being used consistently to its fullest potential to facilitate deep understanding, assessment for learning and 
the levels of intellectual engagement called for by research. While teachers seem to recognize the potential that 
technology has for learning, there are still a number of barriers that restrict effective and appropriate utilization; 
school vision and instructional leadership, time for professional dialogue and team planning, pervasive access to 
technology, and ongoing professional development being the biggest. Teachers want more guidance and support in 
prioritizing the tasks and obligations required in a successful technological implementation in their subject areas. 
Access to computers and various technological innovations is frequently blocked due to filtering methods or 
technological resources are in such short supply that there are simply not enough to provide for each student - i.e., 
laptops, mobile devices. Further, teachers feel they have not been provided adequate professional development in 
the use of the various technological tools and resources, nor have they been given either the time or the training in 
how to integrate technology into their subject area. 

There was very little evidence of change within the school districts over the two years towards building 
their system capacity to advance towards 21st century learning throughout the system. While some districts had 
components of these key features in place in their districts; the challenge for leaders in these districts remained 
finding ways to make all the various components cohere into an integrated, unified whole. The researchers stress 
that a technology initiative implemented over a mere two years is a short time frame to observe the kinds of 
fundamental changes that are called for when intellectual engagement is the goal. That said, we are now into the 
second decade of the 21st century - high school students cannot afford to wait while the school systems translate 
their strategic plans and vision statements about engagement into meaningful, systemic actions and change. School 
systems across the province have an obligation and responsibility to educate students with the competencies they 
need to live well within today’s world. The researchers emphasize that teachers and administrators in this initiative 
were committed to doing a good job on behalf of their students. The majority of them indicated they knew 
engagement was important to student success; however, they were frequently frustrated by what they perceived to be 
system inertia and seemingly insurmountable barriers. 

Educational Importance of the Work 

This study increases understanding of the role of technology in the 21st century classroom and 
understanding of the use of technology for student engagement and success. Findings are significant provincially as 
well as generally to the field of educational technology for illustrating 21st century learning environments that 
utilize technology to increase student engagement, school success and high school completion. The information 
gathered from school visits, classroom observations, surveys and interviews helped to articulate opportunities and 
benefits of extending innovative learning opportunities to all students and teachers in high schools. Case study 
research over two years has informed the provincial ministry and education system of educational practices of 
school administrators, classroom teachers, and the learning of students in technology-enabled high school 
environments. Study findings clearly identify specific barriers and enabling factors that support the types of 
technology enabled learning environments being called for in the current research literature on high school learning. 
This study surfaces both the opportunities and challenges associated with learning, teaching, and leading with 
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pervasive technologies in the 21st century. This research project also contributes to a growing research literature on 
the integration of technology for learning in high schools, and of the ways in which teachers gain technological 
pedagogical content knowledge (Koehler & Mishra, 2008; Mishra & Koehler, 2006). 
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The effects of writing skills on student interactions in online debates 

By Allan Jeong, Haiying Li & Jiaren Pan 
Florida State University 

Abstract 

Because verbal fluency influences perceived competence and credibility, this study identified differences in 
ways students respond to messages posted by students with low versus high writing ability in online debates and 
asynchronous threaded discussions. This study found that arguments posted by students that exhibited poorer 
grammar/spelling elicited 57% more responses that challenged the credibility or merits of the arguments than 
arguments posted by students that exhibited better grammar/spelling. The implications for instruction and future 
research are discussed. 



Introduction 

Collaborative argumentation is an instructional activity for fostering critical discussions in both face-to-face and 
online environments. Argumentation involves the process of building arguments to support a position, considering 
and weighing evidence and counter-evidence, and testing out uncertainties to extract meaning, achieve 
understanding, and examine complex ill-structured problems. This process not only plays a key role in increasing 
students’ understanding but also in improving group decision-making. Online discussion boards are being 
increasingly used to engage learners in dialogue in order to promote more in-depth discussions. However, studies 
show that the quality of online discussions is often shallow. 

Given that argumentation is both cognitive and social in nature, social dynamics can often come into play in 
ways that inhibit or facilitate argumentation. For example, a previous study showed that arguments presented in a 
conversational style (with greetings, emoticons, acknowledgements, addressing others by name) elicited 41% more 
replies that challenged the merits of the argument than arguments posted in an expository style. Challenges 
presented in a conversational style elicited up to eight times more responses providing further explanations for a 
given argument. Studies like this illustrates how certain characteristics of the message that can be intentionally 
manipulated by students can influence how students interact with one another in online debates. 

The purpose of this study was to examine how grammatical/spelling errors influence how likely students 
respond to arguments with challenges and respond to challenges with counter-challenges versus explanations versus 
supporting evidence. The rationale for examining their effects is based on findings from human communication 
research showing that a speaker’s verbal fluency can affect how others perceive the competence, credibility and 
persuasiveness of the speaker (Burgoon, Bilk & Pfau, 1990). As a result, this study hypothesized that: a) arguments 
are more likely to elicit replies that challenge their arguments when arguments are posted by students with low 
writing ability than by students with high writing ability; and b) challenges posted by low ability students are more 
likely to elicit counter-challenges than challenges posted by high-ability students. The questions examined in this 
study were: 

1 . What differences exist in the response patterns to arguments, challenges, explanations, and evidence posted 
by students with low versus high writing abilities? 

2. What differences exist in response patterns produced in exchanges between low-to-low ability students 
versus high-to-high-ability students? 



Method 

The participants were 72 graduate students from four semesters of an online course on distance learning at a 
large southeastern university. Each student participated in four online debates hosted in Blackboard discussion 
forums. Each class was divided into supporting and opposing teams to debate for or against a given claim. Students 
were required to insert a tag into the subject heading of each posting to identify the posting as a supporting or 
opposing argument (+ARG /-ARG), a challenge (+BUT/-BUT), explanation (+EXPL/-EXPL), and supporting or 
counter evidence (+EVID/-EVID) with + and - tags to identify team membership. 
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Each student’s postings was copied and pasted into MS Word and the number of unique grammatical and 
spelling errors was counted. Each student was scored on their writing ability by taking the total number of errors 
(M= 4.96, std= 6.99) observed across all the student’s postings divided by the total number of words (A/=513.39, 
std= 565.8) contributed by the student. The 72 students were rank ordered on writing ability and the median score 
was used to divide the students into low (n=29) and high (n=43) ability group. 

The message tags were modified to identify type of posting and writing-ability group (i.e., ARGH=argument 
posted by high-ability student, ARGL=posted by low ability student). The Discussion Analysis Tool developed by 
Jeong (2005) was used to tally the number of times each type of message was posted in reply to one another to 
generate the frequency and transitional probability matrices below. The right matrix shows for example that 
arguments posted by low ability students (ARGL) elicited challenges (BUT) in 58% of the responses to the 
arguments. In contrast, arguments posted by high-ability students (ARGH) elicited challenges in 41% of responses. 
These probabilities were then graphically conveyed using transitional state diagrams presented below. 
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Results 



The state diagrams show that the response patterns to messages posted by high versus low ability students are 
similar overall. However, the differences in the distribution of responses to arguments posted by low versus high- 
ability students was found to be statistically significant, x 2 (3)=18.1, p= .000. Significant differences were found in 
the mean number of challenges posted in reply to arguments posted by low (M=. 93, std= =1.03, n= 1 12) versus high 
(M=. 59, std=. 82, «=248) ability students, #=-3.30, t//=358, j9=.001. Arguments from low ability students elicited 57% 
more challenges ( ES=+0.18 ) than arguments from high-ability students. The differences in distribution of responses 
to challenges, explanations, and evidence were not statistically significant. 

Additional comparisons showed significant differences in the distribution of responses to arguments 
(X 2 (3)=19.8,j9=.000) and to challenges (x 2 (2)=9.92, p=.001) when examining how low-ability students responded to 
other low-ability students versus how high-ability students responded to other high-ability students. However, no 
significant differences were found in the mean number of challenges posted in reply to arguments posted by low 
(M=. 634, std=. 88, «=112) versus high (M=A59, std=.125, /?=124) ability students, #=-1.665, df=234, p=.091 . No 
significant differences in the mean number of counter-challenges posted in reply to challenges posted by low 
(M=.3\3, std=.639, n= 322) versus high (M=. 267, std=.531, n= 393) ability students, #=-1.057, dj=l\3,p=.29\. 

Discussion & Implications 

Overall, the findings support our predictions that grammatical/spelling errors can affect student-student 
interactions, which is consistent with the research that shows how verbal fluency in face-to-face communication 
affects how people perceive the speaker’s credibility, competence, and persuasiveness. The findings suggest that 
students should not place too much weight on grammatical/spelling errors to ensure that all ideas posted by all 
students are fairly and critically examined. Additional analysis will be conducted to determine to what extent the 
responses patterns are the direct result of grammatical/spelling errors as opposed to the direct result of the quality of 
the ideas presented within each message. Additional findings and the implications for instruction and future research 
will be discussed in the presentation. 
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Statistical Reasoning Skills and Attitude: 

The Effect of Worked Examples 

Li Jin, Yoon Jeon Kim, Marilyn McGhee, and Robert Reiser 
The Florida State University 

Abstract 

The purpose of this study was to examine the effects of worked examples on learners’ statistical reasoning skills and 
their confidence and satisfaction levels. Twenty-six graduate students were randomly assigned to two groups. After 
being provided with the same instructional material on the general rules of comparing group means and how to 
make heuristic inferences based on data without computing a t-test, one group studied worked examples and the 
other group practiced problem solving with no worked examples. A posttest on statistical reasoning skills executed 
after the instruction revealed that participants who studied worked examples outperformed the problem practice 
group (p<.05). The survey on participants’ confidence and satisfaction level indicated that participants who studied 
worked examples were more confident than their counterparts (p<.05) but showed no difference on satisfaction level. 
These results are discussed in terms of how the use of worked examples changed the process of learning of statistical 
reasoning skills. The implications of the use of worked examples in statistics education are discussed as well. 

Keywords'. Statistics education, worked example, statistical reasoning skill, teaching statistical reasoning 

Statistical Reasoning Skills and Attitude: The Effect of Worked Examples 

Statistics education has drawn increasing attention in recent years. First, tremendous numbers of students 
have enrolled in statistics course as a requirement for their degree programs (Garfield & Ben-Zvi, 2005). This is 
because statistical knowledge has become requisite for a wide range of fields of study (Garfield & Ahlgren, 1988). 
Second, there is a strong call for reform of statistics instructions. Most instructors tend to teach concepts and 
procedures rather than statistical reasoning. As a result, students don’t know how to apply statistical concepts and 
rules to concrete problems or make inferences even when they can earn good grades on statistics courses (Garfield, 
2002). Third, there are broad concerns regarding students’ attitude towards statistics courses. Statistical courses are 
viewed by many students as the most difficult in their curricula, which are regarded as obstacles standing in the way 
of attaining their degrees (Gal & Ginsburg, 1994). 

In recent years, more and more educators have advocated that the focus of statistical education should be 
developing statistical reasoning skills rather than letting students memorize definitions, rules, and procedures 
(Chance, Garfield, & delMas, 2000). Statistical reasoning is the way people reason with statistical ideas and make 
sense of statistical information (Garfield, 2002; Garfield et al., 2005). Statistical reasoning may involve connecting 
one concept to another, such as means and spread, or combining ideas about data, such as frequency and chance. 
Statistical reasoning also means understanding and being able to explain statistical processes as a whole, and being 
able to inteipret statistical results, such as explaining the meaning of confidence intervals and significance tests. 
Statistical reasoning skill is one of mostly desired outcome of a statistics course (Garfield, 2002). 

Teaching statistical reasoning is challenging because little is known about the statistical reasoning process 
or how students actually learn statistical ideas (Hawkins, Jolliffe, & Glickman, 1992). The knowledge of how to 
design and implement effective instructional strategies to teach statistical reasoning skills is of importance. However, 
the majority of the studies conducted on teaching statistical reasoning have focused on identifying students’ 
misconceptions and the dynamics of the teaching-learning process. Only a few studies have addressed the aspect of 
the instructional activities and methods that are effective in statistical education. 

Although only a few studies exist, well-designed instructional activities and methods have been shown to 
be effective in facilitating teaching and learning in statistical education. For example, Schwarz and Sutherland (1997) 
and delMas, Garfield, and Chance (1999) reported benefits of using simulations to improve students’ statistical 
reasoning. According to some other authors (Giraud, 1997; Keeler & Steinhorst, 1995; Magel, 1998), small group 
activities appeared to work better for statistics learners when they worked together with 2 or 3 peer students to solve 
problems, discuss procedures, and work on projects. Bradstreet (1996) suggested using real data and questions 
related to students’ experience for instruction because students would understand the methods as they use them to 



105 




solve real world problems instead of memorizing a list of isolated formulas and terms. Bradstreet (1996) also 
suggested using graphics to communicate statistical concepts, procedures, and results. He argued that graphics can 
enhance understanding for intro-level statistics learner. 

While there are many good instructional activities and methods that have been tested as effective in many 
disciplines, they will not necessarily assist students to correctly understand and reason about statistical concepts 
(Chance, 2004). There is a strong need for empirical studies on the effectiveness of utilizing well-established 
instructional strategies in the field of teaching statistical reasoning. 

Learning from worked examples is one of the instructional strategies that have been reported being 
effective in many disciplines (Cooper & Sweller, 1987; Darabi, Nelson, & Palanki, 2007; Darabi, Sikorski, Nelson, 

& Palanki, 2006; Paas & Merrienboer 1994; Sweller & Cooper, 1985; van Gerven, Paas, van Merrienboer, Hendriks, 
& Schmidt, 2003). Worked examples provide an expert’s problem solution for a learner to study. They typically 
include a problem statement and a procedure for solving the problem (Atkinson, Derry, Renkl, & Wortham, 2000). 
According to cognitive load theory research, worked examples, compared to conventional problem-solving approach, 
can reduce the cognitive load imposed on learners. In contrast to the conventional notion of the best way to teach 
problem-solving being to give students a lot of problem to solve, Sweller and Cooper (1985) argue that learning 
from worked examples is a more effective way for novice learners to learn problem-solving. When novice learners 
are presented with problems, they tend to use novice strategies, such as trial and error, which are very demanding on 
cognitive resources. Students learn from worked examples, however, often employ more efficient problem-solving 
strategies they learn from the examples and focus on content of the problems, which makes better use of the 
cognitive resources. Many studies have reported results in favor of worked example instruction rather than problem- 
solving practice (Carroll, 1994; Ward & Sweller, 1990; Zhu & Simon, 1987). 

Considering the nature of worked examples, we think they are good instructional tools for learning 
statistical reasoning skills. Studies show that students tend to respond to statistical problems by substituting 
quantities into computational formulas or procedures without forming an internal representation of the problem 
(Garfield & Ahlgren, 1988) when they work on problem solving practices. This problem can be solved by presenting 
students with worked examples. When presented with worked examples that demonstrate how an expert makes 
sense of statistical data and results, students should be able to leam the rationale for the procedure and how concepts 
can be applied in new situations. 

One other difficulty in learning statistical reasoning skills is that students seem to rely on incorrect 
intuitions and misconceptions to make an inference or judgment no matter how successfully correct conceptions 
have been taught (Tempelaar, Gijselaers, & van DerLoeff, 2006) and those fallacies are very stubborn (Konolde, 
1989; Lecoutre, 1992). One reason why this difficulty is hard to overcome is that students seldom reflect on the 
procedures they go through in making inferences and compare their procedures to those of experts. By studying 
worked examples, students can learn experts’ skills of using correct conceptions and statistical rules in solving 
problems and compare those skills with theirs. We think that worked examples would help students learn experts’ 
statistical reasoning processes and check for errors in their own thoughts. This is likely an efficient way for learning 
complex concepts and skills. 

At the same time, one other important thing in helping students become successful in learning statistics is 
to get them to feel confident about their ability to learn the content and like the content. We think that the use of 
worked example in statistical education would help solve the problem of negative attitude of many of the students. 
Some studies report that students felt more confident in learning when they learned from worked examples (Cripper 
& Earl, 2005; Miller, 2009). Researchers also report that students were better engaged in a worked-example learning 
situation than they were in a conventional problem-solving process (Darabi et al, 2006). Also, students do not like 
statistics mostly because they feel it is hard to learn (Gal et al., 1994). Worked examples are believed to be able to 
reduce cognitive load imposed on learners (make the learning easier) than the traditional problem solving approach 
(Sweller & Cooper, 1995). When students feel less difficult in learning they may be able to feel more confident and 
be willing to learn. 

In spite of the broad adoption of worked examples in other disciplines, no study has been done on the use 
of worked examples in statistics education. In this study, we investigated the impact of worked-examples on 
students’ learning of statistical reasoning and their attitude toward instruction. Specifically, we compared students’ 
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performance and attitude after they learned from worked examples or worked on problem-solving practice. The 
worked examples provided an expert’s procedure of interpreting statistical data and test results and making 
references. For problem-solving practice, participants were required to make their own interpretations and references 
based on data and test results provided to them. Although correct answers were provided to the practice group, no 
details on how the answers were reached were provided. 

Performance was measured by a written test administered after the instructional activities. The test 
consisted of two questions which tested the participants’ ability to make correct interpretation and reference based 
on given data sets and certain statistical test results. Four major points of the reasoning process were examined in 
each of the two questions. Two aspects of the learner attitude, confidence and satisfaction, were measured by a 
questionnaire adapted from Keller’s Instructional Materials Motivation Scale (2009). There were nine items 
focusing on the aspect of confidence and six items examining satisfaction level. 

We expected that students who learned from worked examples would score significantly higher on the test 
and indicate higher confidence and satisfaction level than their counterparts who worked on problem-solving 
practice. These results were expected because worked examples can reduce the cognitive load of the students and it 
would be easier for students to learn from worked example than to develop their own ways of reasoning using 
statistical rules and concepts they just learned. Although correct answers were provided to the problem-solving 
practice group, they needed to figure out the procedure of reasoning or how they were wrong if they could not come 
up with the correct answers. Compared to the latter group, the worked example learners did not need to search for 
appropriate steps for solving the problems and instead they could focus on the content that was important for 
understanding the problems. Theoretically, less cognitive load was imposed on worked example learners which 
would have freed the cognitive capacity for learning effort (Sweller, 1988; Sweller & Cooper, 1985). As a result, 
they would learn better than their counterparts. This result is consistence with many previous studies (Cooper, et al., 
1987; Paas, et al., 1994; van Gerven, et al., 2003). 

We also expected that students who studied the worked examples would report higher levels of satisfaction 
and confidence because learning from worked-examples would make the learning easier (less cognitive load) and 
enhance students’ confidence and performance. As a result of experiencing less difficulty and better performance, 
students would feel a higher level of satisfaction. 

Method 

Participants 

The participants of this study were graduate students at the Department of Educational Psychology and 
Learning Systems at a large south-eastern public university. This department was a part of the College of Education. 
Statistics courses were required for all graduate degree programs in this department. Many of the graduate students 
in this department started to take statistic courses at the first year in their programs. Most of the students 
participating in this study took at least one statistics course or was taking a statistics course. Among the collected 
responses, twenty-six were complete and used for this study. There were thirteen participants in each group. 

Materials 

The material used for the study presented a heuristic way to understand the meaning of a t-statistic used for 
mean difference comparisons. There were six points listed in the material. The first point recalled the knowledge of 
when to use a t-test. The second point described the two parts of a t-statistic — its numerator and denominator. The 
numerator was connected with between group difference and the denominator was explained by within group 
difference. The third and fourth points explained the meaning of the numerator and the denominator of the t-statistic 
in terms of data characteristics (means and standard deviations of the two groups). The fifth and sixth points 
described the situations of significant and non-significant t-test results and their relations with data characteristics. 

Independent variables 

The independent variable was the type of instructional activity used to teach students how to conduct 
statistical reasoning about group difference (t-test) based on data. The first level of the independent variable was to 
study from worked examples (treatment) and the second level was to use conventional problem practice (control). 
The participants in the treatment group were provided with two worked examples on how to make inference about 
group mean differences based on the characteristics of the data without calculating the t-statistics. The participants 
in the control group were provided with two practice problems and correct answers but with no explanations or steps 
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of how the answers were achieved. The participants in the control group were expected to apply the knowledge 
presented in the study material to solve the two problems and compare their results to the correct answers provided. 

Dependent Variables 

Learning was measured using a posttest. The test was designed to assess the participants’ ability to make 
reasonable references about group-difference based on data characteristics without calculating a t-test. Specifically, 
the test presented two scenarios of comparisons of two group means when the means and standard deviations of the 
two groups were provided. The participants were asked to present their reasoning procedures and conclusions on 
whether group means were different based on the information given. The participants’ responses to the tests were 
scored based on the inclusion of important points in the reasoning procedures and the conclusions. One point was 
given for presentation of each of the following three reasoning procedures: consideration of between group 
difference, consideration of within group difference, and comparison of between group difference and within group 
difference. One point was given for getting the correct result. Therefore, a participant got 0-4 points for each 
problem. The highest possible score for the posttest was eight. 

Students’ attitude toward the instructional material was measured by a survey. This survey was adapted 
from Keller (2009)’s Instructional Materials Motivation Survey. The survey used for this study included fifteen 
items, six of which were related to learners’ satisfaction and nine of which were related to learners’ confidence. 
Responses to the items were in the form of 5-point Likert scale. The reliability of the items pertaining satisfaction 
was 0.80 and reliability of the items pertaining confidence was 0.72. 

Procedures 

The participants were randomly assigned to treatment and control groups after they signed the consent 
forms. Then they were provided with the instructional material (printed). The participants in the treatment group 
were provided with two worked examples after the instructional material and the participants in the control group 
were provided with two problems and right answers to the problems after the instructional material. After that, the 
participants took the posttest. At last, they were directed to complete the attitude survey which was attached to the 
posttest. It took each participant about fifteen minutes to finish the task. 

Results 

The outcomes measured in this study were learning and two aspects of learner attitude toward the content: 
confidence and satisfaction. Learning was measured by a posttest which required participants to conduct statistical 
reasoning about group means based on the instructional they were provided in the printing material. A review of the 
data revealed no violation of the assumption of homogeneity variance (F =. 35 for Levene’s test, p= .56). With alpha 
set at .05 and with 13 participants in each group, the probability of detecting a large difference between means 
was .63. 



The mean score on the posttest for participants who studied worked example was 7.5 with a standard 
deviation of 1.20. This was significantly higher than the mean posttest score for participants in problem practice 
group (M=5.3, SD=1.18), t( 24) = 4.78, /K.05. The effect size is .70, which is large according to Cohen (1988)’s 
benchmarks. The results supported the hypothesis that the participants would learn better from worked examples 
than practice on problems. 

Confidence aspect of attitude was measured by a 9-item survey adapted from Keller’s Instructional 
Materials Motivation Survey (2009). The mean confidence score of worked example group (M=4.2, SD= .47) was 
significantly more positive than that of the problem practice group (M=3.8, SD= .49), t(24)=2.18, p<.05. The effect 
size is .41, which is a medium effect based on Cohen’s criteria (1988). The results supported the hypothesis that the 
participants would have higher confidence level when they learn from worked examples than they practiced on 
problem solving on their own. 

Satisfaction aspect of attitude was measured by a 6-item survey adapted from Keller’s Instructional 
Materials Motivation Survey (2009). The mean satisfaction score for worked example group was 3.8 (SD = .57) and 
the mean satisfaction score for problem practice group was 3.6 (SD = .76). Results of a t-test revealed that there was 
no significant difference in satisfaction scores between the two groups, t (24) = .64, p =.53. This result did not 
support the hypothesis that the participants would have more positive attitude toward the content when they were 
presented with worked examples than when they were provided with practice problems only. 
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Discussion 



Supporting the primary hypothesis of the study, the mean posttest score of participants who learned from 
worked examples was significantly higher than the mean score of participants who used a conventional problem- 
practice approach. This is consistent with the results of some studies on using worked examples in other disciplines, 
such as algebra (Carroll, 1994), geometry (Paas, et al., 1994), and calculus (Miller, 2009). The worked examples 
illustrated the step-by-step reasoning procedures for comparing group means. Learners could focus on understanding 
each of the steps at one time. The comprehension of the logic flow from one step to the next is likely to be much 
easier when each step is understood. On the contrary, the problem practice group most likely searched for ways to 
solve the problems based on the statistical concepts and general rules they learned. Especially for novice learners, 
they needed to figure out the relevant concepts, procedures, and steps to process. Thus, learners in the problem 
practice group were probably faced with a higher demand on their cognitive resources than the learners in worked 
example group did. As a result, learners in worked example group grasped the expert way of conducting the 
reasoning with less learning effort while the learners in the practice group came up with novice solutions (in many 
cases with errors) at probably much higher cognitive investment. 

The hypothesis that predicted a positive effect of worked examples on learners’ confidence level was 
supported by the data. The participants who studied worked examples expressed a much positive feeling about their 
capability to learn the content presented in the instructional material than the participants who practiced problem- 
solving. A possible reason for this finding is that when the steps and logic of solving the problem were clear to the 
learners, the learners felt confident about their capability to learn the content and solve the problems. But for 
learners in problem practice group, they were not sure about the solutions they developed. When their results were 
not the same as the correct answers provided, they might have felt difficulty to figure out what was wrong with their 
reasoning process and felt less assured about what they learned. As a result, they feel less confident. 

The hypothesis predicting a positive effect of worked example on learners’ satisfaction level was not 
supported by the data. The participants who studied the worked examples did not feel more satisfied than the 
learners in the problem practice group. This result is probably due to the lack of feedback in the whole process. 
Satisfaction arises from the positive external consequences of a learner’s behavior or the pleasure of having 
successfully accomplished a task (Keller, 2009). The learners in both groups in this study did not receive any 
feedback on how well they performed on the test. Therefore, there was no positive consequence that have may lead 
to boosted feeling of satisfaction. In order to increase learners’ satisfaction level in future applications, we suggest 
that instructors provide learners with feedback on their performance as soon as possible so as to elicit learners’ 
satisfaction while retaining their improved confidence. 

With the large body of empirical studies on the use of worked examples in many fields of subject, we 
believe that researchers and instructors who build upon the present study will help provide a richer picture of the 
benefits worked examples may afford in the field of statistics education. Researchers and instructors may consider 
using worked examples instead of problem practice not only when they teach statistical reasoning skills but also in 
situations that they teach other knowledge in statistics courses, such as distinguishing concepts, choosing 
appropriate tests, and so on. One important suggestion for the use of worked examples is that the instructors should 
provide feedback to the students in a timely and on-target base. In this way, students would be able to accumulate 
their positive attitude towards statistics courses and as a result achieve better learning. 
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Abstract 

The purpose of this study was to examine to what extent each of Bloom’s six levels of cognitive learning outcomes 
were exhibited among four types of postings (argument, critique, evidence, and explanation) in online structured 
debates. Thirty-three graduate students enrolled in an online entry-level course in distance learning participated in 
this study. The threaded discussion postings from four structured online debates were analyzed. The results 
indicated that five of the six levels of cognitive learning (except for knowledge learning) were achieved to a certain 
extent among the four types of postings. There was no significant difference among the frequencies of 
comprehension or application learning across the four types of postings. However, chi-square tests indicated that 
analysis and evaluation were most likely to be exhibited in critique postings, and synthesis was most likely to be 
exhibited in arguments and critiques. Message-response exchanges ending with critiques were most likely to elicit 
higher levels responses and argument postings were most likely to elicit critique responses. The results were 
discussed in light of implications for instructions. Directions for future studies building on the findings of this study 
were discussed as well. 

Keywords : structured online debate, constraint-based online debate, levels of learning, learning outcome, 
online argumentation. 



Introduction 

Structured online debate has been examined as a way to enhance cognitive learning (Jeong, 2004; Jeong & 
Joung, 2007; Moore & Marra, 2005). Structured online debates, also referred to as constraint-based argumentation 
(Cho & Jonassen, 2002) or scripted online argumentation (Stegmann, Weinberger, & Fischer, 2007), are 
instructional activities in online discussions that specify the types of posting students can contribute to online 
discussions, such as claim, warrant, rebuttal, and backing, which are developed based on Toulmin’s argumentation 
model (1958). The use of such constraints has been reported as an effective way to engage students in the processes 
of identifying various perspectives, developing or selecting a solution, and justifying proposed solutions with 
supporting evidence (Kobbe et al., 2007; Cho et al., 2002). Other studies also show that constraints in online debates 
help maintain a task-orientated discussion (Jeong et al., 2007) and support better understanding of the mental 
structures of presented arguments and domain knowledge (Stegmann et al., 2007; Weinberger, Ertl, Fischer, & 
Mandl, 2005). 

Constrained by the types of message that can be posted in online debates, students are engaged in the 
argumentations that require high levels of cognitive processing, such as analyzing, evaluating, and constructing 
(Cerbin, 1988). Students must be able to identify the key points of a statement in order to distinguish between 
claims, grounds, or rebuttals. In organizing lines of statements from both sides of a proposition, the students must be 
able to construct new claims based on multiple ideas. They also need to apply a set of criteria forjudging whether 
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claims are well-supported or warrants are adequately backed. As a result, the students are expected to engage in the 
cognitive processes of higher levels of cognitive learning outcomes in Bloom’s (1956) taxonomy: analysis, 
synthesis, and evaluation (Valcke, de Wever, Zhu, & Deed, 2009). 

Given the promise of structured online debates in facilitating student learning, however, the findings from 
studies that examined the effects of structured online debates have been mixed. For example, Stegmann et al. (2007) 
found that the quality of messages generated by students was increased in structured online debates (script-supported 
discussion) compared to the quality of messages produced by the control group (without scripts). Cho et al. (2002) 
reported that the quality of argumentation during problem solving was significantly improved when students used 
constraint-based argumentation. Students generated more claims (arguments) and grounds (evidences). However, 
the students that did not use constraint-based argumentation produced more warrants. Furthermore, no significant 
differences were found in the quality of individual problem solving performance between the two groups. Similarly, 
Jeong et al. (2007) reported that the use of constraints did not increase the frequencies of messages that presented 
supporting evidence, critiques, and explanations. Moore et al. (2005) reported that students in the less structured 
discussion reached the highest phase of knowledge building while the students in the structured argumentation did 
not. 



The mixed results of the effects of structured online debates are probably due to the different outcomes that 
have been investigated in different studies. Previous studies have used different terms in describing the outcomes of 
structured debates, such as problem solving performance (Cho et al., 2002), frequency of certain types of messages 
(Jeong et al., 2007), or phases of knowledge building (Moore et al., 2005). As a result, it is difficult to consolidate or 
compare the results from multiple studies. Also, since student learning is the ultimate goal for online discussions 
(Schellens & Valcke, 2005), it may be beneficial to examine the outcome of structured debates in a general 
framework, such as levels of cognitive learning as defined in Bloom (1956)’s taxonomy. 

There are six levels of cognitive learning outcomes defined in Bloom’s taxonomy, knowledge, 
comprehension, application, analysis, synthesis and evaluation, sequenced from low to high in cognitive level. In 
knowledge and comprehension learning, learners show recall and understanding of materials, processes, and ideas. 
Application is the use of learned materials (rules, laws, methods, theories, etc.) in new contexts; analysis is the 
breakdown of communication into its constituent elements or parts such that the structures of the idea or the relation 
embedded are made explicit. Synthesis is the ability to reorganize parts together to form a new whole, which 
involves the process of working with pieces from different sources to constitute a new pattern or new structure. 
Evaluation is to judge purposefully and justify the judgment with supports, evidence, data, and reasons. Analysis, 
synthesis, and evaluation are often referred to as high levels of learning. 

Multiple levels of cognitive learning can be achieved in composing each type of postings in structured 
debates. For example, in composing an argument posting, a student should indicate the position he is taking in the 
debate and provide some reasons for his position. Having some knowledge and understanding of the topic and the 
ability to conduct proper analysis are necessary for composing this type of posting. Similarly, when composing an 
explanation posting, a student should be able to show the understanding of the points being discussed, identify the 
relevant information that may clarify his or her statement. In an evidence or critique posting, a student should 
conduct some analysis of existing messages and information from other resources, synthesize multiple elements, 
identify the abstract relations between available information and his position (in support or against), choose the 
preferred one and justify his judgment. Consequently, students are expected to engage in different levels of 
cognitive processing in constructing different types of postings. 

As a result, the use of Bloom’s taxonomy of cognitive learning outcome may enable us to learn how well the 
students learn the content knowledge in structured online debates, based on which we can make inference about how 
well structured online debates may have facilitated learning. This line of knowledge will help educators develop 
insights on whether to use this instructional tool and in what circumstances. For this purpose, knowledge of the 
specific levels of learning achieved in each type of postings may provide further insights into how to improve the 
design and implementation of constraint-based argumentation. Therefore, if we can identify the relationships 
between types of constrains and levels of learning, we may be able to predict which types of messages are more 
likely to elicit higher level cognition. As a result, we can better understand the dynamic association between 
learning and constraints. 
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The purpose of this study is to investigate the learning achieved in structured online debates in terms of the 
levels of learning achieved in each type of postings. In specific, , this study aims to determine to what extent each 
level of cognitive learning has been exhibited within each of the four types of postings used in online debates 
(argument, critique, explain, evidence). The second purpose of this study is to identify which type(s) of message is 
more likely to exhibit a particular level of cognitive learning. A third purpose of this study is to identify what types 
of exchange pairs (e.g. argument -> critique, critique -> explanation) are most likely to elicit or trigger responses 
that exhibit higher levels of learning (e.g. analysis, synthesis, and evaluation). In summary, the questions examined 
in this study are: 

1. To what extent does each type of posting exhibit each level of learning? 

2. Which type of posting is more likely to exhibit each particular level of learning? 

3. What exchange pairs tend to exhibit higher levels of learning? 

Method 



Participants 

Participants were 33 graduate students enrolled in an online introductory course of distance education in a 
larger southeastern public university in the United States. Twenty-two were female and eleven were male. All of 
them were from the program of Instructional Systems. Most of them were distance learners who had a part-time or 
full-time job. All the students consented to participate in this study. Participation in the online debates was 
voluntary and would not be counted toward the course grades. 

Procedures 

Students in the online course participated in a total of four weekly online debates on given topics. The topics 
for these four debates were: media selection and learning, the use of synchronous tools in online courses, cost issue 
of educational technology, and printing versus electronic materials for education. The purpose of the debates was to 
engage students in sharing diverse perspectives of the topics, reflecting on their view points, reasoning based on 
evidence, and supporting or disputing ideas with data. The actual number of participants in each of the online 
debates varied from 12 to 15 students. 

For each debate, the participants were randomly assigned to one of the two teams: supporting team and 
opposing team. They were required to post one of the four types of messages when they support or refute the point: 
argument, evidence, critique, and explanation. Participants were required to insert tags (ARG, EVID, BUT, EXPL) 
into the subject headings of each posting to identify the types of message they posted. Tags had to include a positive 
(+) or negative (-) sign (e.g. +ARG, -BUT) to identify the authors’ team membership (supporting team or opposing 
team). See Figure 1 for a screen shot of the threads and tags in a structured online debate. 
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Instructor Comments 
SUPPORTING ARGUMENTS 

B +ARG1 Sync chats adds more presonal level to learning 
+ EXPL Chats can bring students together 
+EXPL Chats can bring students closer to their instructor 
-BUT Sync chats adds more presonal level to learning 
-EXPLSvnc chats adds more presonal level to learning 
-BUT Sync chat does not necessarily had more personal touch 
RE: +ARG1 Sync chats adds more presonal level to learning 
+ EXPL: Chats build teams 
B +ARG 2 Chats do no exist within a vacuum 
+EXPL: Chats do no exist within a vacuum 
B +ARG3 ProvideReal-timeDvnamic 

' +EXPL Real-timeDvnamicGoodforFeedback 
- BUT VWiat AboutWritinnSkills? 

+EXP3 WtiatAbontWritindSKills? 

-BUT DifferencelsTime 
-EXP3 WhatAboutWritinnSkills? 

+ARG4 BuildsSenseoflmmediacv 
+ARG5 MakesForminqLearninqGroupEasier 



11/7/07 12:35 PM 
11/7/07 12:35 PM 
12/9/07 3:34 AM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
4/23/08 12:09 AM 
4/23/08 12:11 AM 
12/9/07 3:29 AM 
4/23/08 12:15 AM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
11/7/07 12:35 PM 
12/9/07 3:20 AM 
12/9/07 3:20 AM 



Figure 1: Screen shot of threads and tags in a structured online debate 
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Descriptions of what each constraint meant were provided to participants at the beginning of the course. 
Argument was used to make claims of the proposition, convey reasons to support a given position on a given issue. 
Critique was to challenge or critique given proposition, evidence or clarifications. Normally data or theoretical 
supports, or definitions should be included to justify the critique. Evidences were examples, research studies, 
personal observations and experiences that would support the position held by the individual. Explanation was the 
elaboration or clarification of the opinion. 

The rate at which students mislabeled their postings was 13.3%. Mislabeling rate was the percentage of 
postings that were labeled incorrectly over the total number of postings. Mislabeling happened when a student 
labeled his/her posting with a wrong type of constraints (e.g., labeling an argument as a critique), or did not label the 
posting with one of the four constraints. The mislabeling rate for the four debates were 9.3% (7 out of 75), 17.7% 

(14 out of 79), 19% (15 out of 79), and 5.9% (4 out of 69), respectively. The types of posting were corrected before 
conducting the next step analysis. 

Levels of Learning 

Levels of learning exhibited in the structured debates were assessed using Bloom’s (1958) taxonomy of 
cognitive learning outcomes. The six levels of learning from low to high were: knowledge, comprehension, 
application, analysis, synthesis, and evaluation. Each posting in the four debates was coded into one of the six levels 
of learning. A message was assigned the highest level of learning when it contained more than one level of learning. 
See Table 1 for the description and examples of the learning outcomes. A second coder coded one debate and a 
comparison of the codes between the primary and second coder produced an inter-rater agreement rate of 8 1 .4%. 



Table 1: Description and examples of learning outcomes 

Learning Description Example 



Knowledge 



Recall of factors, methods, 
processes, patterns, etc. 



Educational technology refers to the equipments, 
media, and ways we present the instructions. 



Comprehension 



The understanding of the materials 
or ideas being communicated. 



The data suggests as Clark states that, “the 
delivery system affects no inherent difference on 
achievement.” So although I can understand the 
crux of your explanation, I would like to disagree 
with your conclusion. 



Application 



Analysis 



The use of abstractions in 
particular concrete situations. 



...sometimes instructors use these technologies 
when they aren’t really necessary. For example, 
would you really need to see leaves changing 
color for fall on a computer screen... or could you 
just go outside? 



The breakdown of an idea into its 
constituent elements and make 
their relations explicit. 



(Because)Students tend to give increased effort 
and attention to media (such as TV) that are new 
to them. It's not the media that yields achievement 
gains, it's the increased effort and persistence. 



Synthesis 



Working with pieces, parts, 
elements, etc... and arranging and 
combining them into new patterns 
or structures. 



It probably does not matter in cases where the 
objectives to be met are simple and clear. But 
what about cases where there are cognitive or 
psychomotor skills to be learned that are more 
complex, such as learning to perform laser eye 
surgery. Something like this would be difficult to 
learn from a teacher who is just showing pictures 
in a book and reading "how to" steps. 
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Judgments about the merits of 
Evaluation material and methods for given 
purposes. 



As a former Elementary Teacher I can tell you that 
how students think they learn best.. ..and how they 
actually learn best are sometimes two different 
things. If this study was based on students' 
perceptions, is it really a genuine assessment of 
learning? 



Results 

As shown in Table 1, a total of 298 postings has been analyzed and coded. In terms of the number of each 
type of posting, critique is the most, which is 1 14 (M= 28.5, SD = 7.5) and argument posting is the second, which is 
88 (M= 22.0, SD = 5.2). Evidence postings totals 55 (M= 13.8, SD— 6.9) and explanation postings totals 41 (M = 
10.3, SD = 3.4). 



Table 2: Descriptive results of the four debates 



Debate 


No. of 
Participants 




Types of Posting 






Argument 


Critique 


Evidence 


Explanation 


Total 


1 


15 


15 


30 


17 


10 


72 


2 


15 


21 


20 


22 


15 


78 


3 


15 


26 


38 


8 


7 


79 


4 


12 


26 


26 


8 


9 


69 


Total 




88 


114 


55 


41 


298 


M (SD) 




22.0 (5.2) 


28.5 (7.5) 


13.8(6.9) 


10.3 (3.4) 


74.5 (4.8) 



Research Question 1: To what extent does each type of posting exhibit each level of learning? 

Levels of learning exhibited in each type of postings in the four debates are reported in Table 2. No 
knowledge level learning has been observed in any of the postings. The mostly observed learning in argument 
postings is analysis and comprehension, which is 33.0% and 31.8% of total argument messages. No evaluation effort 
has been found in argument messages. 50% of the critique messages in the four debates are analysis in nature. 
Evaluation learning is 7% of critique postings. Comprehension and application are the mostly occurred learning in 
evidence messages with the percentage of 43.6% and 36.4%. No evaluation learning has been observed in evidence 
messages. For explanation messages, analysis and comprehension learning are the mostly exhibited, 39.0% and 
31.7% respectively. 



Table 3: Number counts of messages exhibiting each level of learning in each type of pos tings 



Learning 






Types of Posting 






Argument 
n % 


Critique 
n % 


Evidence 
n % 


Explanation 
n % 


Knowledge 


0 


0 


0 


0 


0 


0 


0 


0 


Comprehension 


28 


31.8 


17 


14.9 


24 


43.6 


13 


31.7 


Application 


11 


12.5 


18 


15.8 


20 


36.4 


9 


22.0 


Analysis 


29 


33.0 


57 


50.0 


9 


16.4 


16 


39.0 


Synthesis 


20 


22.7 


14 


12.3 


2 


3.6 


1 


2.4 


Evaluation 


0 


0 


8 


7.0 


0 


0 


2 


4.9 


Total 


88 


100 


114 


100 


55 


100 


41 


100 



In terms of higher learning levels of analysis, synthesis, and evaluation, critique messages indicate the 
highest percentage of the combination of the three, 69.3%. 55.7% of the argument messages are at the higher level 
of learning. Explanation and evidence messages contain less percentage of higher levels of learning, 46.3% and 
20.0% respectively. 

Research Question 2: Which type of posting is more likely to exhibit each particular level of learning? 
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Table 3 shows the distribution of each level of learning achieved in each type of messages. A chi-square 
test comparing the observed and expected frequencies of each type of posting in specific levels of learning indicates 
which type of posting is more likely to elicit a specific level of learning. We conduct chi-square test for each level of 
learning. 

The chi-square test for comprehension learning shows that there is no significant difference among the 
frequencies of comprehension learning across the four types of messages, /( 3 , N = 82) = 6.68, p = .08. No 
difference is found in application learning outcome across the four types of messages, /T3, N= 58) = 5.86, p = .12. 
However, significant difference is found in the three higher level learning outcomes, y 2 ( 3 , N = 111) = 48.53, p<. 05, 
for analysis, y 2 (3, N = 37) = 29.97, p<.05, for synthesis, and/ 2 (3, N= 10) = 17.20, j9<.05, for evaluation. By 
reviewing the data, we conclude that critiques are most likely to elicit analysis learning, arguments and critiques are 
most likely to elicit synthesis learning, and critiques are most likely to elicit evaluation learning. 



Table 4: Overall distribution of each level of learning in each type of posting (in percentage) 



Types of 






Levels of Learning 






Posting 


Knowledge 


Comprehension 


Application 


Analysis* 


Synthesis* 


Evaluation* 


Argument 


0 


34.1 


19.0 


26.1 


54.1 


0 


Critique 


0 


20.7 


31.0 


51.4 


37.8 


80.0 


Evidence 


0 


29.3 


34.5 


8.1 


5.4 


0 


Explanation 


0 


15.9 


15.5 


14.4 


2.7 


20.0 



*chi-square test for this column is significant at .05 



Research Question 3: What exchange pairs tend to elicit responses that exhibit higher levels of learning? 



Figure 2, 3 and 4 show the transitional relations among different types of messages for low level learning 
responses (comprehension), medium level learning responses (application and analysis), and high level learning 
responses (synthesis and evaluation) respectively. The numbers on the arrowed lines between message types indicate 
the percentage of occurrence for corresponding response pairs. For example, 34% of the observed exchange pairs of 
argument-critique are ended with low level learning (comprehension). 
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Figure 3: Exchange pairs ended with messages exhibiting high level learning (synthesis and evaluation ) 
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As showed in Table 5, thirteen out of the possible sixteen types of exchange pairs have been observed ending 
with higher levels of learning including analysis, synthesis, and evaluation. A chi-square test indicates that the 
frequency of eliciting analysis response in the observed exchange pairs is not equal / 2 (9, N = 94) = 63.66, p <. 05. 
By examining the data, we conclude that argument and critique messages are most likely to elicit responses that 
exhibited analysis. The chi-square test for the equal distribution of exchanges elicited synthesis responses is 
significant, ;f(7, N= 24) = 26.67, p < .05. It seems that argument-critique exchanges are more likely to elicit 
synthesis responses than other exchange patterns. No difference is found for the frequencies of the exchange pairs 
that ending with evaluation learning, ^(3, N= 10) = 4.40, p = .22. This indicates that all of the four exchange pairs, 
which are argument-critique, critique-critique, critique-explanation, explanation-explanation, have the similar 
tendency to elicit evaluation responses. 



Table 5: Frequency of exchange pairs exhibiting higher levels of learning 



Exchange Pair Ended With 


Higher Levels of Learning 


Analysis* 


Synthesis* 


Evaluation 


Argument 


8 


7 


0 


Critique 


57 


14 


8 


Evidence 


12 


2 


0 


Explanation 


17 


1 


2 



*chi-square test for this column is significant at .05 



Post-hoc Analysis 

Post-hoc analysis has been conducted on two emerging patterns from the data. The first pattern observed in 
the data is that the exchange pairs that elicited high level learning and ended with a critique message occur much 
often than other exchange pairs. The chi-square test for exchange pairs that ended with each of the four types of 
constraints is 68.78, with df= 3 and p < .05. This supports our observation that exchange pairs ended with a critique 
message are more likely to elicit higher levels responses. A further investigation of the patterns of frequencies of 
exchange pairs in each of the higher levels of learning indicates that argument messages are most likely to elicit 
analysis responses, / 2 (3, N= 94) = 74.09, p < .05, and synthesis responses, / 2 (3, N= 24) = 28.33, p < .05, but not 
evaluation ,/ 2 (3, N= 10) = 6.80 , p = .08. 

Discussion and Conclusions 

Five out of six levels of learning are exhibited critique and explanation and four out of the six levels of 
learning are exhibited in argument and evidence postings. However, highly uneven distribution of levels of learning 
is observed in different types of constraints. High level of learning (i.e., analysis, synthesis, and evaluation) is 
exhibited in more than half of argument and critique postings. In specific, analysis learning is the mostly exhibited 
in all of the constraints except for evidence postings. Synthesis processing mostly occurs in arguments and critiques 
and evaluation occurs only in critiques and explanations. This indicates that the use of constraints may have directed 
the learning effort toward different levels of learning. 

While levels of learning are distributed unevenly in the four constraints, no knowledge level learning has 
been observed in any types of constraints. This probably is because the nature of the debates. Students are required 
to construct claims to support or argue against given propositions in the debates. Although students must have 
sufficient topic knowledge (Cerbin, 1988), they need to make the decision on what knowledge is relevant to the 
ongoing discussion and make the connections in their postings. This process elevates the students’ cognitive 
learning to at least comprehension level. 

In overall, critique is the most frequently occurred in the debates. This indicated that the participants were 
critical toward the claims made by peers. This finding contradicted with the findings in some other studies (Jeong et 
al., 2007), where the students are unwilling to challenge others. This probably is because the nature of the discussion 
topics and the experience of the students. Most of the students enrolled in this course were working part-time or full 
time when they took this course, they had rich working and social experience with the topics being discussed. 
Therefore, they might have opinions that are very different from one another. 
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As to research question 2, the results shows that students are more likely to exhibit high levels of learning in 
critiques and arguments and less likely to engage in high level processing in constructing evidence postings. This 
may be interpreted as that students provide examples, data, or relevant experiences that may support their positions 
in evidence messages but don’t provide reasons, which makes the postings fact-based. At the same time, analysis or 
elaboration on why these evidences are appropriate may have been presented in explanation messages, which 
explains why over one third of explanation messages are analysis. These results indicate the likely effects of 
constraints on level of cognitive processing. First, argument and critique constraints are very likely to exhibit higher 
levels of learning because the specifications of these two constraints require students to include reasons or 
clarification to support or justify their propositions in the same posting. Second, evidence constraint is less likely to 
engage students in high levels of cognitive processing because the constraints seems to request for factual 
information that are judged as relevant to the issues being discussed. 

For research question 3, the results show that exchange pairs that ended with critiques are most likely to 
demonstrate high levels of learning. It is very noticeable that 80% of the highest level of learning, evaluation, has 
been generated in exchange pairs ended with critiques .This probably is because students are engaged in reflecting 
on ideas, identifying discrepancies, comparing values, and evaluating solutions when they construct critique 
messages. Further investigation of the patterns of exchange pairs and learning outcomes indicates that argument is 
most likely to elicit higher level responses even though the total number of argument postings is not the greatest. 
This may be due to the fact that an argument presented a proposition, which lays the ground for critique and students 
are less likely to respond to one other’s critique with critique and instead with explanation or evidence. 

The post-hoc analysis indicates that students are more likely to exhibit higher level learning when critique 
one other’s statements, and critiques are most likely being elicited by arguments. Based on this finding, we suggest 
that online instructors pay attention to argument and critique postings. More argument postings are likely indicating 
diverse opinions and active participation in the debates. If the number of critique postings is low, it is likely that the 
students do not engage in the debates at a higher level or do not have sufficient experience with the discussion 
topics. Thus, certain interventions should be applied to encourage them to reflect, critic, and integrate ideas. 

In conclusion, the results of this study show that the use of constraints in online debates engaged students in 
the processes of analyzing ideas, identifying relations, making decisions, and constructing new knowledge and as a 
result facilitated student learning. As a way to build up knowledge on the effective use of structured online debates, 
we suggest that future studies should be conducted in different situations and compared to the finding of this study. 
Specific research questions should be addressed in future studies could be the effect of the discussion topics on the 
relations between message constraints and learning outcomes because the students might focus on verifying ideas 
and getting feedback from other on own understanding when they were less familiar with the topics and less 
confident with their statements. Other than discussion content, the characteristics of the students should be 
considered as a covariate in future studies. 
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Abstract 

This study examined three structural attributes of causal maps (total links, temporal flow, outcomes node 
position) and their relationship to map accuracy (ratio of correct/incorrect links) to identify effective mapping 
processes and strategies. The findings suggest that limiting the number of links can increase accuracy where as 
increasing temporal flow was not found to increase accuracy in students’ causal maps and causal understanding. The 
implications of these findings on how to setup causal mapping tasks and directions for future research are discussed. 

Introduction 

To identify root causes to complex problems, causal diagrams can be used to elicit, articulate, refine, and 
improve understanding and analysis of complex problems. Casual diagrams, a network of events/nodes and casual 
relationships/links, have been used to assess systemic understanding of complex problems. However, students’ 
diagrams can vary widely in accuracy if students use different processes while constructing their diagrams. Research 
is needed to determine which processes help students produce more accurate causal maps. 

A previous study (Jeong & Lee, in press) examined how number of links in a map, temporal flow , and lateral 
distance of the outcome node from the left edge of the map ( node distance) were correlated with map accuracy. Map 
accuracy was measured in terms of the ratio of correctly/incorrectly identified root causes and the total number of 
times each student correctly identified the links ( root links) connecting root causes to mediating causes. The study 
produced data to suggest that increasing temporal flow can increase the number of correctly identified root causes, 
and placing limits on the number of links can significantly increase the number of correctly identified root cause 
links. The correlation between distance of the outcome node and causal understanding was not statistically 
significant. However, these findings were based on the maps of only 16 students. 

The purpose of this study was to determine if similar findings could be replicated in a different course, in a 
different lesson, with a larger numbers of students, and with a simpler measure of causal understanding based on the 
ratio of correctly to incorrectly identified links between variables. Like the previous study, regression analysis was 
used to determine to what extent the three variables (number of links, temporal flow, and lateral position of outcome 
node) predicted the accuracy of students’ diagrams following online discussions of the cause-effect relationships 
between variables. The research questions in this study were: 
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1. What are relationships between map these map attributes and students’ causal understandings (ratio of 
correct/incorrect links)? 

2. What is the relative magnitude of each attribute’s impact on causal understanding? 

Method 

The participants were 30 graduate students enrolled in an online course on the topic of Introduction to Distance 
Education at a large university in the southeast region of the U.S. Students were presented with a case study that 
required them to: a) analyze and discuss the possible cause-effect relationships between the six variables/conditions 
that determine how well an organization is able adopt new technology; and b) identify which of the variables might 
be considered root causes to problems with technology adoption. Prior to discussing and debating over the cause- 
effects relationships between variables in an online threaded discussion forum, each student downloaded a map 
template (figure 1) to individually construct a causal map that conveyed their views on how the variables are 
causally inter-related. Students then debated over the merits and logic behind proposed causal links in the discussion 
forum. Based on what they learned from the discussions, students revised and submitted a second and final causal 
map. 
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Figure 1. Map template preloaded with 6 conditions and outcomes. 



Data Analysis 

Causal understanding was measured in terms of the ratio of correctly to incorrectly identified links ( matching 
scores) based on a direct comparison to the instructor’s map. The matching score for each student was computed 
using the jMAP software (Jeong, 2010). Number of causal links was measured by counting all links in each map. 
Ratio of temporal flow was computed by dividing the number of links pointing in the general direction of the final 
outcome node (see green node in figure 2) by total number of link. Node distance was measured by taking the 
absolute distance between root node (starting point of cause-effect relation) and final node (outcome position). 
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Edit Mode Name: student 1 




Figure 2. A student’s map illustrating right-to-left temporal flow. 



Results 

Table 1 shows that with the maps produced prior to discussion, no significant correlations were found between 
attributes and causal understanding. With maps produced following discussion, total number of links was negatively 
correlated with students’ causal understanding or matching scores (/- = -.425, p = .027). 

Table 1 



Correlations between the three causal map attributes and students ’ causal understandings 





Total 


Ratio of 


Node 


Matching 


Variables 


Causal Links 


Temp Flow 


Distance 


Score 


Prior to online discussion (Mapl) 










Total Causal Links 


1 








Ratio of Temporal Flow 


-.099 


1 






Node Distance 


.288 


.302 


1 




Matching Score 

Following online discussion (Map2) 


-.290 


.241 


-.219 


1 


Total Causal Links 


1 








Ratio of Temporal Flow 


.092 


1 






Node Distance 


.481* 


.450* 


1 




Matching Score 


-.425* 


-.303 


-.172 


1 



*p<. 05. **p< .01. 
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Table 2 



The unstandardized and standardized regression coefficients for the variables 



Variables 


Causal Understanding (Matching Score) 


B 


SE 


P 


Prior to online discussion (Mapl) 








Total Causal Links 


-.911 


.927 


.186 


Ratio of Temporal Flow 


.177 


.112 


.300 


Node Distance 


-.038 


.029 


.256 




FQ29)=\.9W,p=\A9 R Square =. 


182 


Following online discussion (Map2) 








Total Causal Links 


-.857 


.346 


.504* 


Ratio of Temporal Flow 


-.112 


.062 


.362 


Node Distance 


.014 


.013 


.234 




F(3,26)=3.043,.p = .049 R Square^ 


.284 



*p< .05. **p<.0\ 



The regression analysis of the maps produced prior to discussions (Table 2) did not produce a statistically 
significant model. The maps following discussions produced a statistically significant model (.F(3,26) = 3.043, p 
= .049) that explained 28.4 % of the variance ( R Square = .284) with a power of 0.75. In this model, total number of 
links had the largest and a negative association with students’ causal understanding (P = -.504, p = .021) after 
controlling for the other two attributes. This finding was consistent with that found in the previous study. As a result, 
the findings altogether suggest that students should be instructed to minimize and limit the number of links in their 
causal diagrams to create more accurate and parsimonious causal maps. 

Although not statistically significant, the model also revealed that temporal flow had a negative association with 
causal understanding. This finding contradicted findings from the previous study. One explanation for this 
difference is that in the previous study the initial position of each node in the map template were positioned along 
the left edge of the template. Hence, students were essentially required to move the nodes to make adequate space 
between nodes so that links could be inserted between nodes. In this study, many of the students did not change the 
position of the nodes most likely because all the nodes were initially placed in the center portion of the template 
with the final outcome node encircled by all other nodes. 

The distance of the final outcome node was found to have no significant impact on students’ causal 
understanding which was consistent with the finding from the previous study. These combined findings suggest that 
positioning the final outcome to one particular side or edge of the map (even though it is correlated with temporal 
flow) is not necessary a critical part or step in the map construction process. 



Discussion 

One main finding in this study was consistent with findings from the previous study. A negative correlation was 
found between total number of links and students’ causal understandings. The instructional implications of this 
finding is that students should be encouraged to minimize and limit the number of links in their causal maps in order 
to help them create more accurate parsimonious causal maps and to increase their level of causal understanding. 
Another alternative is to simply place a limit on the total number of links students can insert into their causal maps. 

In contrast to findings from the previous study, the data showed that temporal flow had no significant 
association with causal understanding. Some plausible explanations for the differences in findings are the following: 
a) in this study, initial position of nodes were placed in the center of the map (not at the far left edge of the map), 
and as a result, many students did not change position of nodes; b) the outcome node in this study was placed in the 
center of all other nodes; and c) students has to consider only six variables instead of 15 variables in the earlier study. 
These findings suggest that: a) the nodes should be placed to the far edges of the map in order to encourage students 
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to consider the temporal flow of events; and b) temporal flow should be encouraged especially when students have 
to examine the relationships between large numbers of variables. 

Given that this was a case-study, the findings reported here are not conclusive because the data cannot be used 
to determine cause-effect relationships. To determine cause-effect relationships between the various map attributes 
and students’ causal understanding, future studies can conduct controlled experiments on each individual attribute. 
Future studies can also be conducted in the following manner to address some of the limitations of this study: a) 
control individual differences in content knowledge & mapping skills, or increase sample size; b) replicate earlier 
study using identical node placements and outcome measures; c) conduct thorough identification of relevant 
variables to facilitate more systematic research; and d) consider other outcome measures (event chains, nodes with 
largest impact). 
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Abstract 

This paper describes an effort to predict alumni success as measured by employer satisfaction in a 
veterinary medical education environment. Due to the complexity of the data, this paper sought to augment 
regression analysis with decision tree analysis to see if the combination of both approaches could result in new 
insights. Potential predictors of employer satisfaction were characterized as either technical or non-technical skills, 
and assigned to smaller conceptual item groupings within one of those areas. The internal consistency of each item 
grouping (concept) was then evaluated using Cronbach’s alpha. In the first approach, regression analysis was used to 
determine how well each concept predicted overall employer satisfaction with graduates. In the second approach, 
decision-tree analysis was used to discover the least uncertain hierarchy of this data, with overall employer 
satisfaction as the outcome. The process and the results from both approaches were examined and discussed along 
with recommended next steps. 



Introduction 

In modern communities, the organization of data is a complex and uncertain task. Often, it is difficult to 
determine how a particular series of events can be parsimoniously represented. Data is now so ubiquitous and 
disparate that meta-analysis has become difficult and ambiguous. This has become a problem in educational and 
research environments, where the sheer quantity of available data can complicate the research/assessment process 
(Fischman, 2011). Nonetheless, effective use of a variety of assessment data holds great promise for the effective 
design/modification of learning environments. 

The goal of this paper is to reflect upon an ongoing, applied method for collecting, organizing and 
analyzing student performance and satisfaction data for the purpose of curricular improvement. A special focus is 
given to the organization of data into categories based both upon the characteristics of the data and based upon the 
ontological structure of the data. We hope to help foster a conversation that will uncover new methods to make 
sense of the increasingly complex and varied sources of assessment data that are available within our communities. 
At the Iowa State University College of Veterinary Medicine (ISUCVM), performance and/or satisfaction data from 
students, faculty, alumni and employers has been collected in order to assess the curriculum. Analysis of assessment 
data at the ISUCVM is used to guide curricular decision making by administrative and teaching faculty. 

One important source of information regarding the success of the curriculum is the satisfaction of 
employers who hire ISUCVM graduates. In this study we sought to determine what graduate characteristics were 
most important in determining whether or not employers would be satisfied with their new graduates. This is an 
important question, because if we can determine what graduate characteristics matter most to employers, we can 
prepare our students with those characteristics in mind. One common approach to addressing this problem would be 
to use regression analysis, with student characteristics as predictors, and employer satisfaction as the dependent 
variable. While this is a beneficial process, we hoped to gain additional insight by also employing decision tree 
analysis. 
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Decision trees categorize data using information gain and have been used effectively within veterinary 
medical situations for making diagnostic decisions (Marsh, W.E., 1993; Ettinger, 2010; Radostits, Gay, Blood and 
Hinchcliff, 2000; Murray and Arguin, 2000; Shumaker, Corso, Rhyan, Philo, Salman and Gardner, 2010). Decision 
trees are also useful because they can organize an arbitrarily large amount of information. Decision trees create 
hierarchical inferences about a data structure through analyzing the uncertainty associated with each concept as it 
relates to an outcome, effectively reducing the entropy within the data structure (Russel and Norvig, 2008). The 
output from a decision tree analysis process is a hierarchical tree that organizes a subset of items into a structure 
useful for making decisions in relation to the outcome(s) used to create the structure. For these reasons, we 
suspected that decision tree analysis would be a powerful tool with which our stakeholders would also be familiar 
(Russel and Noivig; Garcia-Almanza and Tsant, 2008). 

Method 

Since both the internal-consistency/regression and the decision analysis approaches require a well 
organized data structure, our first step was to organize the data into a structure that had both top-down and bottom- 
up validity. Top-down validity comes from a theory-driven, concept-based view of the data, while bottom-up 
validity comes from analyzing the data itself. First we created a hierarchical data structure for all of the assessment 
data used in the ISUCVM by creating theory-based concepts to which questions from any number of instruments 
could be mapped. To create the initial list of theory-based concepts, we gathered a list of concepts from a 
combination of three sources: inter-institutional standards, intra-institutional standards, and more detailed 
interpretations of data by local experts. In the field of Veterinary Medicine, the American Veterinary Medical 
Association keeps a list of core competencies that are useful for maintaining a standardized assessment framework 
across multiple institutions. These inter-institutional standards provided the basic framework for our data structure. 
As with other institutions, the ISUCVM also has its own organizational structure that naturally developed over time. 
Such natural structures can be considered a linguistic technology representing the natural ontology of stakeholders 
and can be incorporated into a data structure to add detail (intra-institutional standards). After applying those two 
sets of standards to our data structure, if there was still ambiguity regarding how data should be structured, we 
approached local experts in the field for organizational recommendations. It is important to note that a data structure 
is not strictly about the natural relationships that occur within our communities. This structure also helps to record 
the relationships between concepts and events in a way that allows each to have a level of agency upon analysis. In 
other words the concepts themselves begin to act as agents in the community, even after the students who produced 
the data are no longer part of it. 

This study involved only a part of the data structure created using the process described above. 

Specifically, we used the results of employer surveys sent to employers of ISU graduates of the classes of 2007, 
2008, and 2009. Each survey asked employers to rate their overall satisfaction with their new employees, their 
overall satisfaction with their new employees’ non-technical skills, and their overall satisfaction with their new 
employees’ technical skills. Employers also rated their employees’ performance in a number of knowledge or skill 
areas. We grouped knowledge and skill areas into categories based on their best theoretical fit. For instance, two 
survey items, “Perform the business related tasks of the position” and “Control expenses and maximize revenue” 
were categorized together into a “Business Skills” item grouping. The main categories included: Data collection, 
Data interpretation. Planning, Taking action. Interpersonal skills. Legal issues, Business skills, Making referrals and 
Problem solving. We then tested the internal consistency of each category using Cronbach’s alpha, and used 
simultaneous and hierarchical regression analysis to determine how well each item grouping predicted overall 
employer satisfaction. That full analysis is described elsewhere (Danielson, Wu, Kirk, Preast and Fales- Williams, 
In-Press), and we will refer to those results when comparing them to the results of the decision tree analysis. 

We used RapidMiner to conduct the decision tree analysis (Mierswa, Wurst, Klinkenberg, Scholz and 
Euler, 2006). For the regression analysis, we were limited in the number of predictors that we could include in the 
analysis by the number of subjects involved. Therefore, the regression analysis only used the 9 conceptual item 
groups (constructs) rather than each individual item. However, decision trees are not limited by the same constraints 
on numbers of participants. Therefore, the decision tree used overall employer satisfaction as the class label (similar, 
conceptually, to a dependent variable). As attributes (similar, conceptually to predictor variables)we used the same 9 
employer item groupings (constructs) that were also used in the regression analysis. In addition to those 9 item 
groupings, we also included all individual sub-items that made up those groupings, as well as three additional items 
from the survey: employer satisfaction with Non-technical skills (overall), veterinary technical skills (overall), and 
veterinary knowledge (overall). The decision tree analysis classified the likely level of employer satisfaction with a 
student, given the other information we have about that student. 
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Results 



The Cronbach’s Alpha analysis suggested that item groupings were internally consistent, with all item 
groupings having a Cronbach’s alpha > 0.7. 

All of the veterinary skills analyzed were significantly correlated with the outcome of employer 
satisfaction. Similarly, all of the non-technical veterinary skills were also positively significantly correlated with 
employer satisfaction. The two aggregate items: Overall Veterinary Skills and Overall Non- Veterinary Skill, along 
with the technical and non-technical skills meta-groupings were then entered into a hierarchical regression analysis. 
The regression analysis revealed that Overall Veterinary Skills alone accounts for over 30% of the variance within 
employer satisfaction. Adding Overall Non-Veterinary Skills to this analysis explained an additional 21% of the 
variance. Using the two meta-groupings in a separate hierarchical analysis, the four technical skills explained 25% 
of the variance and the five non-technical skills together explained an additional 42% of the variance (Danielson, 
Wu, Kirk, Preast and Fales-Williams, In-Press). 

Figure 1 shows the results of the decision tree analysis. In the decision tree analysis, the item Overall 
Veterinary Skills is referred to as ‘Meta-concept: Veterinary Knowledge’. The two items reporting overall employer 
satisfaction with ‘Technical Skills’ and ‘Non-technical skills’ are referred to as ‘Meta-concept: Veterinary Technical 
Skills’ and ‘Meta-concept: Veterinary Non-technical skills’ respectively. The item Overall [Employer] Satisfaction 
was used as the class variable (the outcome being predicted). 



Meta-concept: Non-technical skills 



Control expenses and maximize revenue 



Meta-concept: Veterinary Technical Skills 



Address the legal aspects of veterinary practice 



> 3.500 s 3.500 



VerySatisfied Satisfied 



Identify all problems for a given case 



Recognize when euthanasia is warranted, and carry it out humanely 



Meta-concept: Veterinary Knowledge 



T 



L 



VerySatisfied 



Moderate 



Manage other personnel who work in supporting roles 



> 1.500 * 1.500 





Moderate 



. 

Unsatisfied 



Meta-concept: Non-technical skills 
> 2.500 ^ 2.500 



VeryUnsatisfied 



Figure 1. Decision Tree Analysis 



Discussion 

The decision tree organized information similarly to the regression analysis approach. For instance, the 
decision tree automatically chose the meta-concept of Non-Technical Skills as being the single most informative 
attribute. Flowever, in certain instances, the decision tree chose specific items for classification purposes rather than 
the item groupings from the statistical analysis. For instance, the single item ‘[Ability to] Control expenses and 
maximize revenue’ occurred on the second level of the decision tree as an intervening relationship between satisfied 
and very satisfied outcomes. What is likely happening in situations such as this is that the decision tree is 
purposefully removing redundant information from the tree. Whereas using the traditional approach, we created a 
“business skills” concept as an item-group, the decision tree approach saw the relationship between those items as a 
redundancy and simply reported the one that contributed most to explaining employer satisfaction: Controlling 
expenses and maximizing revenues. Other similarities include reliance upon non-technical skills to make 
distinctions between levels of employer satisfaction. 
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The decision tree analysis gives insight into the data within our community as it relates to employer 
satisfaction. The most informative features are found closer to the top of the tree. For instance, the item grouping of 
Non-Technical Skills does a good job at splitting the data into two subsections where one subsection is likely to 
have high employer satisfaction while the subsection performing poorly in the area of Non-technical skills is likely 
to have overall lower employer satisfaction. Overall, it appears as though items associated with activities that are 
traditionally non-technical in nature perform well as predictive indicators for employer success within the context of 
this analysis. 



Conclusion 

The data structure that was created effectively organized our complex data situation into a structure that 
could be used for the statistical and the machine learning analysis techniques. Using statistical methods, we created 
nine item grouping that helped to simplify the data structure while also quantifying the validity of the resultant 
structure through the evaluation of the internal consistency of these groupings. The use of decision tree analysis in 
conjunction with statistical analysis led to the creation of a minimal entropy classification structure for this data 
source in relationship to these newly created item groupings and the principle outcome of interest, employer 
satisfaction. Overall, the decision trees analysis has been a useful method to for analyzing large data sets because it 
analyzes data automatically whereas approaches such as the statistical analysis used here require predetermined 
inputs. Decision trees also naturally select out redundant information since decision trees use information gain over 
previous concepts as the criteria for adding new concepts to a model. 

Based upon the preliminary reflections of this on-going case study, certain design guidelines can be offered 
to facilitate discovery from data within a community. These guidelines include the creation and use of a relational 
structure that allows for both aggregate and event-based comparisons, such as decision-tree analyses. It is believed 
that data in the format presented will allow valid cross-concept, cross-community comparisons. This belief is based 
upon the way that the concepts were created, grounded in practice and theory, and then separately validated by both 
analytical statistics and information gain approaches. Ongoing and future work will strive to assess the validity of 
this claim through analysis of case studies. It does take time, expertise and money to organize data as outlined 
above; however, we believe that there is potential value added both in terms of new insights and in terms of 
increases in curricular quality that may offset this initial investment of resources. 
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Abstract 

Our research explored an online social networking site centered around diabetes health maintenance to describe how 
knowledge and participation is shaped by the informal social interactions within the site. Social network analyses 
were conducted to visualize patterns of interaction among participants and to characterize membership based on 
centrality to the community. Qualitative content analyses of discussion threads within the social network revealed 
patterns of knowledge sharing and guided participation. 

Theoretical Background 

Educational researchers have begun to explore how people learn in a range of informal contexts, where 
free-choice learning is the dominant focus, and which involves participation within social networks of families, 
hobbyists, or other communities (e.g. Bell et al., 2009; Dierking & Falk, 1994). Learning in informal settings can be 
framed within sociocultural perspectives^ where learning is viewed as participation within a community of practice 
(Lave and Wenger, 1991) and where cognitive shifts in individual reasoning are closely linked with the social 
processes that are part of the sociocultural setting (Nasir, 2005). Thus, learning can be considered as participation in 
everyday, situated activities that arise from interaction with material and social resources (Barab & Roth, 2006; 
Rogoff, 1995). 

Online communities can be either created by design or emerge in more self-organized ways because of 
their function in the world (Barab, 2003; Barab & Roth, 2006; Hoadley, 2010). Broadly characterized as social 
network sites (identified by Boyd & Ellison, 2007), online communities offer individuals the opportunity to establish 
an identity within a bounded context and interact with others who share similar interests (Gunawardena et al, 2009). 
These activities closely resemble Wenger et al’s (2002) definition of a community of practice, as well as the broader 
notion of “networks of practice” (Brown & Duguid, 2001), which are extended networks of weak relational ties that 
share knowledge across organizational boundaries. The communities we are exploring foster networks of weak 
relational ties solely online, differentiating it from notions of community with sustained histoiy or common heritage 
(Barab & Duffy, 2000) and they are more similar to the notion of “affinity spaces” as characterized by Gee (2004). 
Affinity spaces, an alternative to the notion of community of practice, are characterized as spaces where individuals 
interact around shared interests, affinities, or goals, rather than as members of a community. Such spaces can serve 
as significant knowledge repositories, distributing knowledge across a community, and providing important “social 
affordances” that maintain and reproduce knowledge tied to practice (Brown & Duguid, 2001; Gee, 2004). 

Our research explores how everyday, online networks of practice or affinity spaces create opportunities for 
participation and knowledge sharing that are quite different from those encountered in formal educational settings. 
Social networks form interconnected webs of identities, experiences, and expertise, with participation focused on 
sharing of tacit and formal knowledge and practices of a domain, profession, hobby, or interest. In areas such as 
health literacy, such networks offer unique opportunities for disease-related information to be compiled and shared 
beyond one individual (Frost & Massagli 2008), leading to the growth of “Flealth 2.0 and “Medicine 2.0” (Flughes, 
Joshi, & Wareham, 2008). Indeed, Web 2.0 technology has broken down geographical and cultural barriers to 
facilitate new forms of participation in specialized groups and practices that would otherwise be difficult to identify 
and support without institutionalized access. Our research looks closely at the nature of a network of practice to 
more fully understand how it functions to share, guide and advance knowledge and participation. 
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Research Questions 

Our research examined the following questions: 

1 . What are the patterns of participation in the online social network? 

2. What types of knowledge are shared within the online social network? 

Research Context and Methodology 



The Online Social Network 

Our research examined a health social networking site that facilitates discussions about managing diabetes. 
The site is associated with a major non-profit organization that is dedicated to research, fund raising, and 
management of diabetes. Within this online social network, users post questions, share knowledge and practice, and 
provide stories and contextualized advice related to dealing with the disease on an everyday basis. Users connect 
with each other primarily through creation of users profiles, signature files with relevant disease -related information 
(e.g., the types of medicines they take), and discussion forums focused around topical categories and user diagnoses. 
For instance, users can contribute to topical forums such as “exercise and fitness” and “diabetes technology and 
equipment”, and they can also join forums based on their affiliation with a particular aspect of the disease (e.g., 
gestational diabetes, Type I, or Type II diabetes). Users can post questions and answers, internal links to other 
threads within the site, external links to other source material, and photos. 

Data Collection and Analysis 

We focused our analyses on the “Type 2 Diabetes” forums. We selected this forum due to the large number 
of postings to it (over 70,000 total posts). We transcribed 100 threads over an 18-month period, selecting only those 
that exchanged questions and information related to knowledge and strategies for managing the disease; personal or 
off-topic discussions were not selected for analyses. 

We used a descriptive research design to observe participant discourse without influencing it. We used two 
complementary analysis techniques: (1) social network analyses to provide quantitative metrics of member 
participation; and (2) content analyses of the interactions to identify patterns of knowledge sharing and guided 
participation practices. Details of each analysis are described below. 

For the social network analyses, 9 threads were selected for both types of analyses, which were based on 
representativeness of a variety of types of posts as well as diversity of authors of new threads (no repeat authors). 
The threads were parsed to identify the initial poster and responders, and we created an adjacency matrix to 
represent and record the ties between actors (in this case, the individual community members) in the network. This 
matrix was then imported into UCINET software to visualize the data and to run subsequent analyses on the 
structure and composition of the network. 

For the content analyses, we selected 9 threads for content analysis, which together resulted in a total of 
169 discussion board posts for analysis. We began with a process of open coding, to identify emergent themes and 
patterns. Our first step was to select the unit of analysis (Babbie, 2007), and we chose individual messages within 
each discussion thread as the unit of analysis, since it allowed us best to answer the question about types of 
knowledge that are shared within the network. Two researchers jointly coded the first 4 threads and as a process of 
this initial round of coding, we generated a list of categories and definitions for the categories. We coded the 
remaining 5 threads individually and then jointly resolved discrepancies. This second round of discussion resulted in 
refined category definitions and applications. 

Content analyses yielded the foundation for our characterization of the types of knowledge that are evident 
in the dialog and are shared within the OSN (research question 2). Underlying our analysis is an assumption of the 
OSN as an “affinity space” (Gee, 2004) with features that encourage sharing of the following types of knowledge: 

(a) individual as well as distributed knowledge, or knowledge distributed across people, mediating devices (tools), 
and materials and to which people can network their own individual knowledge; (b) dispersed knowledge, or 
knowledge that resides outside of the OSN site; (c) specialized (intensive) as well as broad (extensive) knowledge 
related to the domain of interest; and (d) tacit knowledge, or knowledge developed through practice that may not be 
explicitly codified or stated as such. These features provided a broad theoretical lens for examining in more detail 
the specific types of knowledge evident within individual posts. 
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Results and Discussion 



Question 1 : What are the patterns of participation in the online social network? 

To identify patterns of participation in the site, we used social network analyses (Katz, Lazer, Arrow, & 
Contractor, 2004) to analyze connections between the various actors (i.e., members of the online community) as 
well as the overall grouping of actors within the network. Our initial analyses utilized the entire set of actors (n=194) 
associated with 9 discussion threads. Figure 1 is a sociogram of ties between the entire set of actors within the 
network. A second partition of the data was done using the k-cores algorithm, which stratifies groups of members 
based on the number of connections each member has, and in this specific case, each member is connected to at least 
8 members (this number selection was automatically performed by the UCINET software that we used for analyses). 
The sociogram (see Figure 2) that thus emerges is more readable since it indicates the presence of groups based on 
varying levels of connections. The actors highlighted with the red color are those that have the highest number of 
connections and thus are more ‘core’ to the network. 
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Figure 2: Sociogram indicating k-cores (Legend: red = 8 connections, lime green = 7, gray = 6, pink = 5, blue = 4, 
dark green = 3, black = 2) 

To further investigate this network, we decided to focus on individual actors within the main k-core group 
and understand how they are connected to others. We approached the participation aspect from two angles. First, we 
generated ego networks, i.e., a sociogram for individual actors that shows how they are connected to others in the 
network and to whom they are directly connected. Based on the k-cores (and our content analyses) we selected lizz, 
Morr, and alan as good candidates, since all three of them have a central position within the network. The 
sociograms (Figures 3, 4, and 5) indicate that with their central positions in the network, lizz and Morr have very 
similar types of connections with other members, while Alan’s network looks different. 




Figure 3: lizz ego network 
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Figure 5: akin ego network 

The ego networks of Lizz and Morr are dense and share many actors in common, while Alan 's network is 
less dense and not as connected. The centrality values of these three actors (Table 1) indicates some interesting 
trends about their prominence in the network, in terms of the number of people they are directly connected to 
(degree centrality), the number of people they connect in the network (betweenness centrality), and the in-degree 
(how many messages are directed to them), and out-degree (how many messages do they send out). 



Name 


Degree centrality 


Betweenness 

centrality 


In degree 


Out degree 


lizz 


65 


2137 


26 


62 


Morr 


60 


1434 


23 


57 


alan 


62 


1990 


30 


59 



Table 1: Centrality measures 



All three actors seem relatively prominent in terms of people to whom they are directly connected, but their 
betweenness centrality indicates the lizz can be considered the biggest ‘broker’ in terms of connecting people to each 
other. Despite alan’s comparatively smaller and sparser network, he is second in connecting people, while Morr 
connects the least amount of people in the network. It is also interesting to note that akin has the largest number of 
messages directed to him and is comparable to both lizz and Morr in the number of messages he sends out. Our next 
step is to parse the reasons for the differences in the betweenness of these actors, since these differences indicate that 
whom one is connected to affects the power and prestige of the person in the network. I.e., it is likely that lizz has a 
higher betweenness centrality because she is connected to other well-connected actors (or actors with high 
betweenness centrality) within the network (e.g., Knoke & Yang, 2007). We also intend to use these additional 
analyses to focus our attention on the central members in the network to identify the types of interactions, expertise, 
and knowledge that they share. 

SNA for individual threads 

We also performed a second analysis on individual threads of discussion to identify the patterns of 
participation within each thread and how members interact with each other. We selected two of the longest threads, 
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with 16 and 56 responses each. Figure 5 represents the sociogram for thread 1 (with 16 responses) and Figure 6 
represents thread 2 (56 responses). 




As is evident from this sociogram, the initial poster, dsta, gets direct responses from multiple actors within 
the network. There isn’t too much interaction among other actors, except between lizz and Jami (indicated by the 
weight of the ties) and which suggests a relatively strong sub-conversation. There is also evidence of another 
question being posted within this thread, as visible from the multiple single responses to Wend. 




Figure 6: RTYFIF1 thread 

The second thread, in contrast to the earlier thread, has a more interconnected structure. While there are 
multiple direct responses to the initial poster ( arka , as indicated by the number of ingoing arrows), the sociogram 
indicates that a number of sub-conversations occur featuring mary, gran, and lizz. There is also evidence of 
connections between known actors - for example there is an outgoing line from gran to lizz to Morr, which indicates 
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that gran is calling on lizz, who in turn calls on Morr. We also see evidence that actors respond directly to posts of 
interest, without necessarily responding to the initial poster — example, Talv responds to yell and abue,\ust as dave 
responds to my 3k. and xena responds to /nary and lizz. A number of interpretations can be drawn about the patterns 
of participation based on these individual threads: 

a. Posters focus on responding to the initial poster’s concern primarily 

b. Members may call on other members with more expert standing or pertinent information. 

c. Longer threads have more complicated response patterns that can signify that multiple issues are 
being addressed within the context of a single thread. 

d. Individual actors respond to specific posts only either out of interest (i.e., they seek to clarify 
something) or expertise with the topic (i.e., they seek to contribute new information or 
perspectives). 

We will continue SNA of other long threads to identify further types of interpretations that can be drawn from this 
data. 



Question 2: What types of knowledge are shared and distributed within the online social network? 

To examine knowledge-sharing practices in the online social network, we identified recurrent patterns of 
discourse that emerged from the discussion boards. The online postings were divided into two primary patterns. The 
first codified our interpretation of the original poster’s purpose for starting the thread. Participation in the 
community was goal-directed, active, and focused around various intentions of members. We identified the 
following overall trends to describe the types of goals or purposes apparent in users’ initial posts: (1) to seek 
specific solutions to a problem; (2) to seek advice, feedback or explanations around an experience, plan, or goal; (3) 
to share a resource or story presumed to be of interest or relevance to the membership at large. In most instances, 
original posters contextualized their post through story telling or data sharing (e.g., glucometer readings). 

The second primary pattern we identified related to knowledge sharing within the dialog focused on the 
types of responses provided by members of the community to an original poster’s thread. We identified three main 
categories of knowledge sharing among community, which are presented in Table 2, along with sub-categories and 
definitions: (1) Articulating knowledge or understanding; (2) negotiation or extension of knowledge or 
understanding; and (3) reified practice-based knowledge or understanding. 

Articulating knowledge or understanding 

This category was evidenced by dialog that was informative in nature. Types of responses coded for this 
category included posts that served to give advice, answer a question, provide an opinion, confirm or oppose a plan, 
decision or idea, or share a resource or finding. This pattern of dialog was often evident in initial posts that asked 
fairly structured or specific questions (e.g., how do I decide what glucometer to purchase?) or solicited specific 
opinions or advice from members. 

Negotiation or extension of knowledge or understanding 

This second category involved more elaborated responses. Typical responses within this category provided 
extensive elaboration of facts or explanations, introduced new terminology of the domain, directed attention to 
needed areas of learning, and provided alternative framing of a problem or context beyond what was acknowledged 
by the original poster. Responses coded as this category of knowledge sharing were marked by efforts to help a 
member solve a problem or to bring to bear new considerations or perspectives on a problem. 

Reified practice -based knowledge or understanding 

This category was marked by sharing of knowledge rooted in personal or community practice. By 
interacting within the social networking space, members pass on their practical knowledge to each other through 
joint activity (working on solving a problem together) or by explicitly sharing routine personal practices, rules of 
thumb, and reflecting on lessons learned from related personal experiences. The result is a rich repository of stories 
and practices that are available and reified for consideration in certain types of situations. Practice-based knowledge 
is considered to be spread and appropriated in instances where members acknowledge their intent to use a specific 
practice or acknowledge trying out a practice recommended by another member. 
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Table 2: Categories, descriptions, and examples of knowledge sharing practices in the OSN 



Coding Category 
Category 1: 

Articulating knowledge 
or understanding 



Category 2: Negotiation 
or Extension of 
Knowledge or 
Understanding 



Category 3: Reified, 
Practice-Based 
Knowledge or 
Understanding 



Descriptions, Codes, and Examples 

Description : Articulating thoughts, opinions, and ideas; such responses lead to a 
more refined discussion of the topic or problem presented. 

Category 1 Codes : shares information; confirming; opposing; gives advice; gives 
opinion 

Examples : A poster asks for recommendations about blood glucose meters. 
Community members give advice, eg., “ There are a lot of good meters out there 
but the best one is the one that your insurance covers the strips for. Most 
companies will give you a meter for free knowing that you have to use their 
strips to have it work,” and opinion, e.g., “....If you get to pick your own and you 
have good insurance and low co-pays I personally like the Accu-Chek Aviva 
Meter. 1 like the strips on the AVIVA meter because they are not as small as they 
are with the Beyer and One Touch strips. The strips are much easier to get a hold 
of...” 

Description : Knowledge sharing that extends perspectives available for 
consideration around a problem or topic of interest, typically by providing new 
resources, explanations, observations, or focal points for new learning. 

Category 2 codes : questioning/requesting clarification; provides elaboration; 
provides explanation; alternative framing; referring to new background or SIG 
file data; directing attention to new area of learning; and introduce new 
terminology of the domain. 

Examples : An initial poster asks for an explanation for what happens during a 
night time low. Community members supply new terminology, e.g., “ Sounds to 
me like you did go very low during the night. By the time you "woke up" and took 
your reading @ 136, your liver had made an adjustment for you on its own. 
Called a 

liver dump,” provide alternate framing, e.g., “ Really hard to say if you went low 
because you did not test at the time in question. Bren has outlined one scenario 
that might explain it, but it could be entirely unrelated to Diabetes, or blood 
sugar level,” and provide explanation, e.g., “If you drank more than usual, that 
combined with insulin can cause low blood sugar, and delay the liver dump that 
brought you back up again, perhaps for hours.” 

Description: Knowledge sharing rooted in the sharing of personal and 
community-at-large practices. 

Category 3 codes : presenting routine personal practices; codifying “rules of 
thumb”; summarizing others’ experiences or knowledge of the domain; 
expressing intent to alter practice. 

Examples : (Routine personal practice): “/ keep a meter and a can of coke by the 
nightstand [in case my blood sugar goes low during the night]. It is nice if I just 
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test and then drink my handy coke and then go back to sleep.” (Rule of thumb): 
“A good rule of thumb is to test right before bed and if the number is 100 or 
lower eat a snack. This will help you to not go low overnight. You might also 
want to do some middle of the night testing to see just how your numbers are 
running in the middle of the night.” 



Across all patterns of interactions, we found that storytelling was a main component of the discourse in the 
OSN. For original posters, the main function of the storytelling was to contextualize a problem or situation in a way 
that points to a need for a change in practice. For respondents, storytelling reflected a variety of functions, 
including: (a) sharing lessons learned from one’s own experiences that point to recommendations for another 
members’ situation; (b) providing counter-evidence to interpretations or recommendations of other community 
members; (c) illustrating expert practices or procedures; and (d) as a means to personally connect with users. 

Data sharing (e.g., fasting blood sugar levels) was often used as a tool for communication about the 
problem/goal context as well as for giving advice and defining criteria for successful practice. In some instances, 
participants were able to share data in response to adopting a new practice recommended by a community member, 
leading to a collective process of testing and refining explanations about the source of a problem. 



Discussion and Implications 

The findings from this study may contribute to the growing literature on lifelong, informal learning (Bell et 
al., 2009). Our research investigated an online social network where people voluntarily go to learn and to participate 
in a shared goal of disease management. This research may extend what is known about self-organized, online 
networks of practice for everyday learning (Brown & Duguid, 2001). 

One of the primary insights of this study involves a more detailed description of how participation and 
knowledge sharing is structured in online contexts to support individuals’ efforts at self-initiated learning via 
appropriation of social resources available through the social network (Barron, 2006). In these spaces, interactions 
serve a guiding function towards advancing knowledge and practice of both individuals and the community as a 
whole. Consistent with Rogoff s (1995) notion of guided participation, a framework rooted in sociocultural theory of 
everyday learning, our data reflect goals and events of everyday life, where participants mutually engage in co- 
constructed, practical activity. Through content analyses, we found that participation in the community was goal- 
directed around intentions of members. Guidance was either tacit or explicit, including deliberate attempts to 
instruct, provide rules-of-thumb, or link to authoritative resources. It also involved more incidental comments, 
opinions, or experience sharing. Guidance was mostly provided by experienced members and served the following 
functions: (1) to encourage or discourage an idea, plan, interpretation; (2) to extend or re-frame a stated problem, 
goal, or understanding to one that is more consistent with the norms of practice; (3) to advance or “upskill” 
participants’ practice by providing alternative concerns, observations, explanations, strategies, or interpretations of a 
situation; and (4) to reify knowledge, experiences, lessons learned, or rules of thumb for newcomers to use in their 
own contexts. These findings, along with those derived from our social network analyses, point to a significant role 
of experienced members in the community. Future research should investigate more closely the characteristics of 
these experienced members and the guiding roles they play within the community. We also see a need to identify 
productive vs. unproductive discourse patterns to better inform design of OSN’s. 
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Introduction 

According to Jonassen (2000), the 21 st century requires people to apply reasoning processes to solve complex 
problems (e.g., making predictions, inferences, drawing implications, articulating explanations). These processes are 
usually applied in learning tasks to achieve conceptual understanding and to perform problem solving. Jonassen 
(2000) proposed using causal maps to facilitate the reasoning process because causal maps enable learners to 
visualize the causal relationships between variables and then articulate, explore, and assess their understanding of 
complex problems. In recent years, computer-supported mapping tools have increased in numbers: CmapTools 
(Ifenthaler et ah, 2009), Ulysse (Desthieux et ah, 2010), STELLA (Schecker, 1994), and jMAP (Jeong, 2009). 
However, a limitation of these studies is that students were not given empirically tested guidelines on how to draw 
more accurate causal maps. Few studies have examined and tested what processes lead students to construct higher 
quality maps. The purpose of this study was to determine what behaviors and sequences of behaviors performed 
during map construction might lead students to create better causal maps. As a result, the research questions 
addressed in this study were the following: 

1 . What behavioral patterns exist when students are constructing causal maps? 

2. How do sequential behavior patterns differ between students that created more accurate maps and students 
that create less accurate maps? 



Method 

Participants 

In fall 2010, 18 graduate students (3 males, 13 females) enrolled in the Learning Cognition course at a large 
southeastern university. Students were taken from programs of Elementary Education, Counseling Psychology, 
Instructional Systems, and Foundation of Education. 

Procedures 

Students were given 50 minutes to apply concepts and principles from reinforcement theory to produce a causal 
map conveying the optimal way to sequence stimulus, responses, and response contingencies to achieve desired 
behavioral changes. To draw the maps, students used jMAP - a computer-based mapping tool developed by Jeong 
(2010). The entire processes of drawing causal maps were recorded using a computer screen capture program. 
Students’ causal maps like in Figure 1 were assessed by matching identified links and link attributes (e.g., direction, 
impact, contingence) with those in an expert map. 
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Figure 1. Example of student’s causal map 

Sixteen videos were produced to capture students drawing sessions (two missing data due to technical 
problems). For this case study, the video recordings of three students that scored high in accuracy and three students 
that scored low in accuracy were analyzed to identify potential differences in map construction behaviors. 

Sequential Data Analysis 

Each student’s actions were coded into any one of the six following categories: 

N : make a change on the initial position of a node 

R: make additional changes to position of a node 

S: simultaneously shift position of a group of nodes 

L: create a link between nodes 

A: change attributes (impact, direction, probability) of a causal link 

C: change a link to point from one node to another 

The sequential data was coded and aggregated into one data file for students who created maps of high accuracy 
and another file for students who created maps of low accuracy. Using the Discussion Analysis Tool (DAT) 
developed by Jeong, the sequential data was converted into frequency matrices to reveal how often each action was 
immediately followed by another action, probabilities and z-scores were then computed (Table 1) to determine the 
probabilities in which one action was performed immediately following another action and whether or not the 
observe probability was significantly different from the expected chance probability. Transitional diagrams for each 
group (Figure 2) were then produced with DAT to visually convey the observed transitional probabilities. 
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Table 1 Frequency matrices and transitional probabilities matrices 
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Figure 2. Transitional diagrams for the high (top) and low accuracy group (bottom) with significant probabilities 
highlighted in yellow. 
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Results 



Some differences in behavioral patterns can be observed when we compare the two state diagrams in Figure 2 
and trace through the sequences of possible actions based solely on the transitional probabilities that were found to 
be significantly above chance and/or expected probabilities. The transitional diagram of students with high accuracy 
maps shows that when this group of students changed the initial position of one node, they were very likely to 
follow that action by changing the initial position of yet another node and so on (62%). The next possible action that 
was most likely to follow was the creation of links to connect one node to another node (11%). Once a link was 
created, the next likely action was to create yet another new link (57%). Therefore, a typical behavioral sequence 
performed by this group might look like I-I-I-L-L-L. In contrast, the diagram for students with low accuracy maps 
shows that they began adding links to connect nodes only after they had re-positioned a node on their map (1 1% of 
the time) as opposed to changing the initial position of a node. Furthermore, the low accuracy students repositioned 
nodes a total of 36 times, where as the high accuracy students repositioned their nodes only 18 times. And once they 
started adding a link, these students were 20% much more likely to continue adding more links (77%) in comparison 
to the high accuracy group (57%). 

These noted differences in behavior patterns may suggest that high accuracy students may alternate more 
frequently between positioning nodes and adding links between nodes during the map construction process than low 
accuracy students. This type of behavior seems to suggest that high accuracy students may be using a more 
systematic and step by step approach to temporally and/or sequentially sequence one node after another node. In 
doing so, each time a node is positioned and linked produces constraints on what subsequent causes or effects should 
be considered, moved into positioned and linked to the current map. In contrast, the low accuracy students tendency 
to move and reposition nodes more extensively followed by heavy activity creating and linking nodes suggests the 
possibility that these students did not use the visual spatial (and possible temporal) organization of the nodes to help 
them isolate and identify the correct cause effect relationships between the nodes. Given that temporal flow in causal 
maps tends to be found in maps of higher accuracy than in maps of lower accuracy (Lee & Jeong, 2010), these 
differences in behavioral patterns might serve to explain the observed differences in map accuracy. 

Implications for Future Research 

This study presented valuable insights into students’ behaviors when drawing causal maps. For the future 
research suggestion, a large number of cases of student’s map drawing demonstration should be analyzed to produce 
more reliable frequencies of students’ behaviors. The future research also may examine correlations between each 
probability of sequential behavior and key elements in causal maps such as total number of links, temporal flow and 
node distance. Students’ prior knowledge should be controlled to avoid possible threaten of internal validity. Finally, 
the cause effect relationship between observed processes and their effects on map accuracy must be tested in 
controlled studies by requiring students to use specific processes and by comparing the results of using one 
particular process over another process. 
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Abstract 

This paper describes a metric called “performance efficiency,” and its use in cognitive task analysis. This 
metric provides a means of determining the learning efficiency of instructional conditions. Performance efficiency 
will be described in the context of recording technologies that are often used in software usability studies. While 
usability is often considered in the programming of software environments, the “learnability” of these environments 
is more the concern of instructional designers. The advantages and disadvantages of these types of metrics and 
methodologies will be discussed in detail. Thus the purpose of this paper is to consider the applications of the 
“performance efficiency” metric to the design of instructional materials. 

Introduction 

Educational researchers have used a medical model to develop instructional materials; that is we hope to 
design instruction, which is both efficient and effective (Lewis & Barron, 2009). Gagne (1964) was one of the 
earliest educators to describe these two general categories of dependent variables. He proposed most educators are 
concerned with (1) “the rate of attainment of some criterion performance” (efficiency) and (2) “the degree of 
correctness of this performance” (effectiveness) (Gagne, 1964, p.295). It is an underlying theme of this paper that 
when these variables are applied to the design of instructional materials, we are considering the “learnability” of the 
instruction. 

Nielsen (1993) defined usability by developing several subcomponents (learnability, efficiency, 
memorability, errors, and satisfaction). Soloway, Guzdial, and Hay (1994) called for Norman’s “user-centered” 
design philosophy to be more “learner-centered.” Nielsen’s (1993) definition of learnability “How easy is it for users 
to accomplish basic tasks” is a subjective measure of “perceived usability,” rather than a more objective measure. 
However, Nielsen (2001) proposed we should consider the user’s opinions and suggestions, but only after actually 
watching them work with the software. That is we must start by observing learners before considering their 
perceptions. It’s interesting that if we were to look to the international standards organization (ISO) for a definition 
of usability, we would find that they also chose to use Gagne’s variables (ISO 9241-11, 1998). 

While cognitive load may not seem related to usability, similarities reveal themselves if you consider the 
measures underlying this theoretical framework. Cognitive load theory is an instructional theory that is concerned 
with the learnability of instructional materials. This theoretical framework has become quite influential within the 
field of instructional design (Ozcinar, 2009; Paas, van Gog, & Sweller, 2010). Cognitive load theory is primarily 
concerned with procedural knowledge, task performances and problem-solving. Cognitive load measures are a 
combination of subjective data (mental-effort ratings) and performance scores (Tuovinen & Paas, 2004). These 
measures have been found to be reliable and correlated with error rates (Ayres, 2006) but not all cognitive load 
theorists agree with the use of subjective measures, and have proposed we consider more direct or objective 
measures (Briinken, Plass, & Leutner, 2003; Whelan, 2007). This concern has led to the impetus for this paper and 
the “performance efficiency” metric described in the next couple of sections. This type of research (task analysis) 
has a rich history and is certainly a metric to be used in cognitive task analysis. 

Task Analysis 

Task analysis researchers have used observation or photography/videography as a means of data collection 
for decades (Clark & Estes, 1996; Gilbreth & Gilbreth, 1917). Some of the earliest task analysis studies were made 
with stopwatches and the newly developed technology of chronocyclegraphy (Gilbreth & Gilbreth, 1917). This was 
the use of long exposure photography, which allowed for the detection of movements over time. While the Gilbreths 
were early pioneers of time motion studies, even they were aware of the underlying rationale for this type of 
research. They state it when they say “that the learner shall be taught the best way immediately, that is, from the 
beginning of his practice” (Gilbreth & Gilbreth, 1917, p.82). So they were amongst the first to promote efficient 
instruction, for it allows a learner to be more efficient with their time, and simply leam more. 
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In the 1970s and 80s, researchers began to realize cognitive processes controlled the behavior of those 
performing tasks (Clark & Estes, 1996). So it was then, when Psychology took a major step forward to develop 
cognitive task analysis. The Cognitivists analyzed task performance, but were aware of the decision making 
processes that occur during problem solving (van Merrienboer, 1997). Therefore cognitive task analysis takes in to 
account these cognitive processes, to subdivide complex tasks into their component parts in order to support a 
learner’s performance (van Merrienboer, 1997). 

Early researchers developed stepwise processes for analyzing learner behavior as they interacted with 
computers (Clark, Feldon, van Merrienboer, Yates, Early, 2008). Card, Moran, and Newell (1983) produced a 
seminal work in this arena as they developed a form of cognitive task analysis called GOMS. A GOMS analysis 
produces a step-by-step text-based description of the procedural knowledge required to accomplish a task (Card, 
Moran, & Newell, 1983; John & Kieras, 1996a). GOMS is an acronym which divides the components of computer- 
based problems and learner actions into goals, operators, methods, and selection rules. The GOMS model analyzes 
computer-based problems by reducing them to solution steps, known as problem solving operators (Dix, Finalay, 
Abowd, & Beale, 2003). 

Even before GOMS existed, Merrill (1971) had proposed instructional designers use an “information 
processing" perspective toward task analysis as they design instructional materials. Later, researchers found that a 
GOMS analysis resulted in positive modifications to instructional materials (Elkerton & Palmiter, 1991; Steinberg & 
Gitomer, 1992; Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, Clark, 2008). Therefore there is certainly 
precedence for the use of GOMS analyses in the design of instructional materials. One reason for this is because 
experts often have difficulty articulating exactly how they perform procedures (Villachica & Stone, 2010). This type 
analysis allows for the instructional designer to work with a series of subject matter experts and synthesize their 
input to develop a more comprehensive product (Sullivan, et ah, 2008). Indeed several studies have found empirical 
evidence that support the efficacy of CTA-based instruction (Clark et ah, 2008). It is with this thought in mind that 
researchers are beginning to consider cognitive task analysis within the context of cognitive load theory (e.g. 
Shachak, Hadas-Dayagi, Ziv, & Reis, 2009) for they can be used together within a GOMS-like analysis (Lewis, 
2008). It is in this way that a cognitive task analysis may provide the means to test the predictions of cognitive load 
theory. 

Efficiency and Effectiveness 

So what is needed is a way of producing instructional design guidelines that are based on empirically 
driven studies. Researchers need an objective method of assessing and evaluating instructional materials which 
1) is based upon Soloway’s learner-centered design philosophy (Soloway et ah, 1994), and 2) allows researchers to 
analyze instructional strategies, to find those which are the most efficient and effective. Cognitive load theory has 
provided us with several constructs that compare instructional conditions (Briinken, Seufert, & Paas, 2009; 

Tuovinen & Paas, 2004). Paas and van Merrienboer initially developed the primary cognitive load construct, 
“relative condition efficiency” (Paas & van Merrienboer, 1993; Sweller, van Merrienboer, & Paas, 1998). Relative 
condition efficiency combines learner performance scores and mental effort ratings to compare instructional 
conditions (See Equation 1). 



Relative condition efficiency — 



Z -Z 

Performance MentalEffort 



V2 



(i) 



The resulting data is graphed on a biplot (See Figure 1) to allow researchers to compare the relative 
efficiency of instructional conditions. Since its development, relative condition efficiency has become an important 
basis for much of cognitive load research (Paas, Tuovinen, Tabbers, & Van Gerven, 2003). However, this 
measurement relies on indirect or subjective mental effort ratings (Briinken, R., Plass, J. L., & Leutner, D., 2003). 
Paas and van Merrienboer (1993) were aware of this limitation and even state in their original article that this 
construct should be qualified with performance data. While “relative condition efficiency” is a measure of the 
cognitive efficiency of an instructional condition, because it uses mental effort ratings, it does not include time in the 
equation. Therefore it is difficult to truly analyze the efficiency of the learner’s performance given an instructional 
condition. 



While learners are able to reliable report their perceptions of the instructional conditions (Ayers, 2008), it 
would be helpful to truly be able to gauge learner performance over time. Learners become more efficient each time 
they solve similar problem solving scenarios, as they acquire the underlying problem schema. Lewis (2008) 
synthesized each of these ideas to develop another construct, called “performance efficiency.” Performance 
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efficiency is similar to relative condition efficiency, blit uses performance time rather than a mental effort rating 
(compare Equations 2 and 3). 
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Performance efficiency was developed to complement rather than compete with relative condition 
efficiency. Therefore it is hoped that this separate efficiency metric may be used to strengthen cognitive load 
research and to analyze instructional conditions in a similar way. Performance efficiency contrasts instructional 
conditions in much the same manner as relative condition efficiency, to combine Gagne’s dependent variables in a 
biplot (see Figure 2) with group performance times and performance scores. 

A generalizable research methodology 

Generalizablity is perhaps the most valued aspect of any task analysis methodology (Crandall, Klein, & 
Hoffman, 2006). The methodologies employed by the Gilbreths were generalizable and still useful today. Software 
task analysis studies are a modification of those early time motion studies, but now aimed at computer-based 
environments. Software packages like Techsmith Morae (Techsmith, 2011) are used to record user interactions with 
software. These software solutions can be used to compare the learnability of instructional conditions (Lewis, 2008). 
These computer-based solutions are a generalizable methodology for conducting a cognitive task analysis. Like the 
time motion studies of the early 20 th century, today’s computer based task analysis studies use time as a dependent 
variable. Certainly recordings have been used for years, but recordings of a learner’s onscreen actions, during 
computer-based training is relatively new. Recordings free the researcher from the constraints of time, allow 
researchers to review, categorize and analyze the learner’s performance. If necessary, they can watch a learner as 
they perform some behavior repeatedly, in order to document multiple aspects of that behavior (Lewis, 2008; Martin 
& Bateson, 2007). This is because a researcher can easily rewind the recordings to document multiple outcome 
variables, which may have occurred simultaneously. Thus researchers do not have to document learner behaviors as 
they occur, because this methodology allows these researchers to review learner actions weeks or months after the 
actual behavior. 

Gagne’s variables may be measured with recording software like TechSmith Morae (Techsmith, 2011). 
Learners are given credit for completing a problem solving operator (Lewis& Barron, 2009). To tally completion of 
a problem solving operator, researchers may use a rubric or checklist based upon the problem being solved (See 
Tablel). In this table the problem solving operators are listed in the table to the left and then the operator is checked 
in the table cells (usually with a point value). This is made easier by using a spreadsheet application to tabulate 
scores automatically. Finally, performance time may be measured in seconds with the recording software. This 
software allows a researcher to mark the timeline of a learner’s onscreen recording and to designate the beginning of 
a performance, in this case with the first initial mouse movement. Once data points are found per learner then a large 
spreadsheet of data points may be made. For more information on calculating the metric values consider other more 
detailed sources (Clark, Nuguygen, Sweller, 2006; Lewis, 2008). 



Conclusions 

The most important implication of this research is that it produced a new generalizable metric for 
contrasting instructional conditions. This metric considers performance outcomes as they relate to the design of 
instructional materials. This is as opposed to the cognitive efficiency of the materials which is the role of relative 
condition efficiency. It is hoped that these metrics can be used together to promote cognitive load research. This 
performance efficiency metric was an outcome of a cognitive task analysis, and a development of a dissertation 
study (Lewis, 2008; Lewis & Barron, 2009). An important implication of this study was that it demonstrated a new 
methodology for analyzing both learner tasks, and on-screen behaviors of the learner. This analysis provided 
instructional design researchers a generalizable methodology of evaluating learner responses (documenting learner 
errors & problem solving operators) within a computer based learning environment. 
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Recording learner onscreen actions is generalizable to any e-learning environment. It uses “off-the-shelf’ 
software (Techsmith Morae) to document and analyze learner behavior. Along with the performance efficiency 
metric this methodology allowed for a comparison of instructional conditions. Each of these can be used together 
with cognitive load measures as a research methodology to compare and contrast instructional materials. Task 
analysis researchers have used observation (via photography/videography) for decades, this is yet another 
methodology. It is hoped that others make uses of these methods and find them useful as a means of improving 
instructional materials. 
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Figure 1. A biplot of Relative Condition Efficiency 
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Figure 2. A biplot of Performance Efficiency 



152 



Table 1 



Picnic problem accuracy rubric 



flip 


layer 

*** 


rotate 




*** 




*** 


*** 




*** 


*** 


*** 


*** 


*** 




*** 


*** 




*** 






*** 


*** 


*** 


*** 


*** 


*** 


*** 


*** 


*** 


*** 






*** 


*** 


*** 


*** 






*** 


*** 


*** 


*** 


*** 




*** 


*** 




*** 


*** 


*** 


*** 


*** 




*** 


*** 




*** 


*** 




*** 


*** 




*** 


*** 


*** 


*** 


*** 





0 0 0 



move item 

umbrella 
tshirt 
head 
right leg 
head 2 
purple shirt 
hat 

s left leg 
bent right leg 
left leg 
green shorts 
arm 2 
pink shorts 
left arm 
body 

picnic basket 
arm 

right arm 

torso 

table 

bird3 

bird2 

bird 1 

0 0 



153 




A Developmental Study of an Instructional Systems Design Model for 

Elementary School Teachers 
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Introduction 

Instructional design or instructional systems design conceptualizing the preparatory process of instruction 
is the most representative field in educational technology (Gustafson, 1981, 1991). Instructional design is defined as 
a systematic approach of analyzing, designing, developing, implementing, and evaluating instruction (Gustafson & 
Branch, 2002). Although a variety of Instructional design models have been emerged, the Dick & Carey model 
(Dick, Carey & Carey, 2005) is one of the most frequently referred conceptual models. 

However, even the Dick & Carey model is confronted with the various critiques both theoretically and 
practically. The Dick & Carey model is appeared to have a limitation of premising linear process (Jones & Richey, 
2000). The outputs from one component of the model become the inputs for the next component and the final 
outcome can be revised by conducting formative evaluation. Thus, opinions of stakeholders cannot be adequately 
reflected until the last component. To overcome this limitation, the Rapid Prototype model is being explored as an 
alternative. This approach emphasizes the importance of opinions of stakeholders by developing the prototype in 
early phase. 

In addition, the Dick & Carey model has a problem of not being well applicable in schools. Selected studies 
reported that school teachers apply the instructional design models rarely and irregularly although they recognize the 
importance of prescriptive activities of the model. According to Park (2007), though school teachers are aware of 
developing instructional skills and needs analysis as the most important things, defining instructional object based 
on needs analysis and developing materials are not fulfilled well. It is because they don’t have enough time and 
knowledge. Also, they think those processes are unnecessary because teachers’ guide is enough to prepare the 
instruction. Jeong (2009a) also supported these results. He reported that elementary school teachers did not practice 
instructional design well, and they thought the instructional design as a formal, meaningless, and difficult activity. 
That is, Dick & Carey model as one of representative Instructional systems design models is not well utilized in 
schools. 

Instructional design is on the top of the practical areas of teachers and functions as a significant factor to 
become a professional teacher. In spite of its importance, teachers do not apply the Instructional systems design 
models because they did not adequately reflect the conditions and situations of the school. 

The purpose of this study is to develop an optimal Instructional systems design model for elementary 
school teachers considering the progress of recent theoretical development and reflecting the situation and 
conditions of school realities. In order to achieve this goal, an initial model was developed by suggested principles 
by reviewing relevant literatures and then revised it to the optimal condition by design-based research methodology. 

Theoretical Background 

The Theoretical Problems of Instructional Systems Design Models 

Instructional systems design is suggested to apply scientific principles of various learning and education to 
activities teachers should consider when they plan instructions (Gagne, Briggs, & Wager, 1992). This approach, 
stressing systemic of problem solving, is called as systematic instructional design (Dick, Carey, & Carey, 2005), 
instructional systems design or instructional development. Though instructional systems design model have worked 
satisfactory in guiding instructional design process systematically, there are some theoretical limitations. 

Dick & Carey’s model (Dick, Carey, & Carey, 2005) representing the instructional systems design model 
has linear process. In traditional instructional design or instructional systems development, instructional designers 
can get information provided learners’ situational contexts at the end of a long development process (Naumann & 
Jenkins, 1982). As a result, learners’ needs are not applied exactly and the outcomes become unsatisfactory despite 
of spending lots of time and costs (Lim & Yeon, 2006). To overcome this problem, Tripp and Bichelmeyer (1990) 
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suggested rapid prototyping which emphasize the rapid development of final form in the early part. This model was 
based on previous instructional systems design models and each design activity is reiterate and simultaneous. It 
emphasizes the continuous interaction between designers and a client and this makes the designers possible to reflect 
the client’s demands fully. 

In addition, as traditional instructional systems design models has been influenced by theories from 
behaviorism and cognitivism, they are not reflected recent research of constructivists. To improve this problem, we 
can consider scaffolding, portfolio, and rubric based on scientific research results stressing learner centered learning 
activity or constructivism. Portfolio is not just learning outcomes, but useful to reflect learning processes. Rubric is 
helpful to improve an achievement as the guidance suggesting learning goal or expectation level of task (Noh, 2008). 

Last but not least, instructional systems design models so far are too generous to use in specific design 
situations. Since initial model of Dick and Carey (Dick & Carey, 1978) was proposed, lots of models have been 
developed. However, there are not clear differences except using just a little bit different terms. It is necessary to 
define instructional systems design components applicable to specific contexts. Not only that, it is also asked to use 
refine process and gather data systematically to improve model’s efficiency. Consequently, more realistic and 
specific instructional systems design model considering teaching context and condition is required. 

Instructional Design Conditions of Elementary School Teachers 

Instructional design is the core of all professional activities of the teacher (Young, Reiser, & Dick, 1998). 
Recently, the needs of teachers’ competency of instructional design grow bigger with emphasizing their instructional 
expertise. Though they also recognize the importance of instructional design activity, they don’t practice it properly 
(Park, 2007). 

The researches about instructional planning show that teachers don’t follow rational objectives-first model 
(Driscoll, Klein, & Sherman, 1994; Reiser, 1994). They plan instruction focusing teaching contents (Bullough,, 

1987; Clark & Peterson,, 1986; McCurcheon, 1980; Shavelson, 1983), and establishing learning goal is not 
importantly considered in their design (Clark & Peterson, 1986; Shavelson, 1983; Yinger, 1980). Clemente and 
Martin (1990) identified the primary gap between teachers’ instructional planning and instructional design model. It 
is because teachers used mental plan not written plan when they planned instruction. Moallem (1998) found that in 
teachers’ thinking and their instructional model, planning, practicing, and evaluating steps were interlocked. He also 
said that teachers do not follow instructional systems design model when they plan instructions but used well 
developed mental structure with several years’ teaching and learning experiences. The Young et al. (1998) also 
reported that teachers did not emphasize the defining learning goals and do any test based on the goals. 

In Korea, although most of the teachers recognized the importance of prescriptive instructional systems 
design activity, they did not practice it fully(Park, 2007). The reasons were as follows: lack of time and cost, lack of 
expert knowledge, dependence on teachers’ guidance, lack of school supports, and regarding it unnecessary. 
Meanwhile, more reasons were pointed as follows: national curriculum(Karaca, Yildirim,, & Kiraz, 2008), lack of 
teachers’ recognition, several socio environmental factors(Clemente & Martin, 1990; Park, 2007), and cultural 
differences in design (Kircshner & Van Merrienboer, 2002). 

As discussed above, in spite of the importance of the prescriptive instructional systems design activities, 
model has not been well practiced in real-world settings. This is correspondent to the result of several studies that 
model is not reflect the reality. Therefore, we would like to develop the instructional systems design model 
considered existing state of elementary school teachers’ instructional design. 

Meanwhile, we also identify four primary conditions of elementary school affecting teachers’ instructional 
design activities. 

First, in elementary school curriculum, there exist basic contents of textbook to teach. Since contents are 
already provided, teachers do not carry out needs analysis, task analysis, learner analysis, and environment analysis 
(Park, 2007). Dick and Carey’s model (Dick, Carey, & Carey, 2005) is asked to do those analyses in the beginning 
and then develop instructional strategy, and those processes are distinct to the model. 

In addition, the primary outcome of the instructional design activity is units design. Teachers’ guides 
suggest lesson activities and materials by units. The units are usually composed of similar contents and goals to 
teach and central to design instruction. 

Third, there are lots of resources and data to refer for designing instruction. Digitalized data and materials 
not only by national organizations but also by private firm are provided. These resources are positively or negatively 
influenced elementary school teachers’ process of instructional design (Han, 2009). 

Last, school and classrooms are equipped with information communication technology. Computers 
connecting internet and projection TV electronic boards and others are ready to use in the classroom. 
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As discussed, previous instructional systems design models have several theoretical problems and are not 
used well in a real setting because they do not reflect elementary school teachers’ situations and conditions. 
Therefore, this study seeks to develop an optimal instructional systems design model for elementary school teachers 
by considering the progress of recent theoretical development and reflecting the situation and conditions of the 
school realities. 



Methods 



Design-Based Research 

This study is conducted as a design-based research process. Design-based research method, being 
developed by Collins (1992) and Brown (1992), is focused on real educational environment where participants 
interact with one another, and on design settings rather than in laboratory settings. Within design-based research, 
iterative cycles of design and implementation are used, and each implementation is used as an opportunity to collect 
data for the subsequent design. By doing so, it can be illustrated and predicted when and how the learning occurs in 
real educational setting (Kim, 2008). 

The purpose of design-based research is to generate competency for knowledge product or perfect 
understanding of certain area. The research is divided into two types such as product and tool research and model 
research (Richey & Klein, 2007). Since this study has focused on developing instructional systems design model for 
elementary school teachers, this study should be considered as a type of model research. The opinions of the 
elementary school teachers’ about model based on theories are collected and analyzed in depth. After that, those 
results are applied to revise the previous developed model repeatedly. 

Research Process 

According to the design-based research, the model is developed and revised based on expertise’s review 
and elementary school teachers’ opinions. 

The development of the initial model 

The initial model was developed including steps and specific principles based on the literature review. The 
reviewing the literatures were conducted in two areas; one was about the limitations of current instructional systems 
design model and the other was related to the reality of using instructional design model in elementary school. 
Through literature review, ‘development of user-centered rapid prototype’, ‘conducting realistic and iterative 
formative evaluation’, and ‘applying principles of constructivism’ are identified. In addition, following issues are 
suggested; ‘conducting analyzing learning objective as the first step’, ‘developing the model based on unit design’, 
‘benchmarking proficient teachers’ instructional design process’, ‘using the resources efficiently’, and ‘emphasizing 
evaluation plan’. These principles were applied to develop the initial model. The guidebook with specific examples 
was also developed to increase the possibility of applying the model to instructional design in schools. 

The development of the second model based on the experts’ review of the initial model 

The professors concerned with elementary education and experienced elementary school teachers reviewed 
the first model. Through semi-structured interview, we gathered the data and analyzed it. A packet of the 
information including guide, abstract of the model, and questions those we would ask before the meeting was 
emailed to each of the reviewers. The interviews were conducted to investigate the improvements of the model and 
the participants’ responses were analyzed. The responses were categorized according to model in general and steps 
and then coding them based on improvements. The results were reviewed and applied to refine the initial model, and 
the second model was developed. 

The development of the final model based on the review of the second model 

To increase the possibility of utilizing the model in elementary school, the model was tested for use by 
elementary school teachers and the opinions were suggested. A total of 6 teachers, 3 novices under 2 year’s 
experiences and 3 experts with over 6 year’s experiences, participated. The expert teachers were selected to collect 
opinions about the instructional design process and the novice teachers were selected to investigate the difficulties 
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when they design the instruction according to the model. Another packet including guide, abstract, and review points 
was electronically sent to the reviewers. The interviews were conducted on August 16, 2010. First, description of the 
purpose of the study and a brief review of the model were conducted. After that, 6 elementary school teachers were 
divided into two groups, novice and expert, and then asked to design instruction according to the processes of the 
model. They followed the steps from 1 to 7 except the last step of summative evaluation with using worksheets 
developed to aid teachers’ design of instruction. To identify the improvements, interviews were conducted with 
them after completed the steps of the model. The participants’ responses were organized according to model in 
general and steps. The second model was refined based on the participants’ feedback and the final model was 
developed. 



Results 



The Development of the Initial Model 



The initial model was designed by considering these two criteria: the direction of the theoretical 
development of the instructional design model and the possibility of applying the model in the elementary school. 
<Table 1> shows the design principles identified by reviewing the relevant literature. 


<Table l>Design principles derived from literatures 


Implications of researches about 
instructional design model 




Design principles 


Limitation of linear processes 
(Naumann & Jenkins, 1982) 




Rapid prototyping method, formative 
evaluation 


Effect of learner-centered learning activity 
(Park & Min, 2008) 




Instructional strategy design to maximize the 
learners’ participation 


Deficiency of model applying in school 
(Park, 2007) 




Specialized model(for novice elementary 
school teachers) 

Considering school context 



Above all, existing instructional design model couldn’t reflect learners’ need exactly because evaluation is 
conducted at the end of the process (Naumann & Jenkins, 1982). Therefore, based on the rapid prototyping method, 
‘formative evaluation’ steps are applied after unit and lesson design. Second, in constructivism, the core of the 
instructional strategies is learner-centered learning activity. It means that learner leads learning activity according to 
development level and trait of the learner (Duffy, Lowyck, Jonassen, & Welsh, 1993). Since elementary school 
students are in the concrete operational period, effective learning occurs when a variety of cognitive activities are 
suggested (Piaget, 1972). That is, the focus is designing learning activities to learn through cognitive participation. 
In this regard, instructional strategies based on constructivism, learners’ self-evaluation and teachers’ reflection, are 
suggested. 

Third, the elementary school teachers use the instructional design model rarely because the user of the 
model is not defined and the previous instructional models are not reflected the conditions and situation of the 
school (Park, 2007). The initial model was developed based on the understanding of school context and elementary 
school teachers’ reality. 

<Table 2> shows the design principles from the literature review about teachers’ instructional design 
activities. 



<Table 2> design principles elicited from literatures about teachers' instructional design 



Implications from 
‘real condition of teachers’ 
instructional design’ 




Design principles 


Already existing of learning objective and 
contents to teach in national curriculum 
(ministry of education, science and 
technology, 2008) 




Beginning from analyzing learning object 


Emphasis on design for unit (Song, 2001) 




Lesson design after unit design 
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Experienced teachers’ instructional design 
method 

(Cho & Yun, 2009) 




Experienced teachers’ formative 
evaluation on novice teachers’ 
instructional design 


Abundant instruction-learning materials 
(Jeong, 2009a, 2009b) 




Emphasizing on analyzing instruction- 
learning resources 


Insufficient plan of evaluation 
(Park, 2007) 




Emphasizing evaluation planning 


First, teachers usually focus on instructional strategy because learning objective and contents to teach 
already exist (Bullough, 1987; Clark & Peterson, 1986; McCurcheon, 1980; Shavelson, 1983). Also, elementary 



school teachers don’t have to analyze learners and learning environment since they can identify learners’ learning 
level by observing everyday in the classroom. Considering the context of the elementary school, analyzing learning 
objective and contents is the first step based on teachers’ implicit analyzing learners and learning. Second, unit is 
composed of similar contents (Byun etl, 2007). Designing a lesson after designing a unit first, is helpful for an 
effective learning because the relation among learning contents can be identified. 

Third, if novice teachers can learn how experienced teachers design an instruction, they may reduce their 
trial and error (Chung, 2001). Experienced teachers’ instructional design methods including instructional strategy, 
instructional materials, and lesson plan can be shared via the internet (Kim, 2008). The analyzing of instruction and 
learning material including these data can have a role to point the right direction of instructional design for novice 
teachers. Experienced teachers’ know-how can be applied to formal evaluation of novice teachers’ instructional 
design. Though the subject of formative evaluation should be students, it is not effective to collect elementary school 
students’ opinions because of elementary school students’ cognitive development. In this circumstance, most 
effective and realistic way of evaluation is from experienced and competent coworkers. In addition, using instruction 
and learning materials which are uploaded in the internet affect teachers’ instructional strategies. Since elementary 
school teachers must teach minimum 4 hours every day, the existing resources are helpful to teach. Therefore, 
analyzing of teaching resources is stressed and the materials should be revised based on the specific needs for 
students. 

Fourth, in the process of teachers’ instructional design, planning the evaluation is not conducted well (Park, 
2007). It is important to check of students’ learning outcomes, so planning evaluation is highlighted. 

Based on the design principles, the initial model was developed (Figure 1). 
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Figure l.The first instructional systems design model for elementary school teachers 



The characteristics of the initial model are as followings. First, design a lesson after design a unit first. By 
identifying a unit objective first and then design a lesson, the lessons can be more organized. Second, analyzing 
learning objectives and contents is the first step of designing instruction processes. This is because educational 
objectives are given to teachers in the elementary school context. Third, formative evaluation is rapidly conducted 
after designing of a unit and a lesson. This leads to revise the prototype of the instructional design through 
experienced teachers’ evaluation not waiting the end of the process. Fourth, analyzing instructional resources is 
conducted prior to design instructional strategies. The analyzing instructional resources and designing instructional 
strategies are not separate process but incorporated process. Analyzing instructional resources places above 
designing instructional strategies is because there are plenty of resources and they can influence to design 
instructional strategies. 
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The activities in each step are as follows. The first step, ‘analyze learning objectives and instructional 
contents of units’, is about analyzing unit objectives and contents based on the learners and learning environments. 
Teachers investigate the unit contents by reviewing ‘teachers’ guide’ and ‘textbook’. Then, they revise the learning 
objectives and contents according to learners and learning environments. They can plan to integrate lessons or 
subjects and change the orders of the lesson. The outcome from analyzing is shown to be ‘unit structure map’, 
relationship of learning topics in unit. 

Second step is ‘design instructional strategy for units’. After making a ‘unit structure map’, teachers plan 
the teaching method roughly. Considering students’ characteristics and level, learning places and learning materials 
are selected. The ideas are written on the ‘unit structure map’. 

The ‘plan evaluation’, the third step, is the process of deciding evaluation method based on the instructional 
strategies and media. Assessment criteria for performance evaluation should be checked and conduct evaluation 
right time. 

Forth step, ‘conduct primary formative evaluation’, is for evaluation of unit design. The evaluation is 
conducted by experienced teachers in the same grade. The comment of experienced teachers about unit design can 
be applied to revise the instructional design of the unit. 

‘Analyze instructional resources’ is the fifth step. After finishing unit design rapidly, teachers confirm the 
learning objective again and write learning objective using behavioral verb. And then, they search for the 
instructional resources to teach effectively and select among them. When they select the material, they must 
consider learning objective, students’ learning level, and possibility of using it. 

The sixth step is ‘design instructional strategies for lessons’. In this phase, teachers plan the detailed 
instructional strategies and write the lesson plan. They plan learning activities, learning order, and organizing 
environments on the basis of analyzing learning task. When they design the learning activities, learners’ 
participation, feedback, and opportunity of reflections should be considered. The form of lesson plan is based on the 
Gagne’s nine events. 

The next step is ‘conduct secondary formative evaluation’. It is consisted of experienced teachers’ 
evaluation, rehearsal of teaching, conduct implementing a lesson, students’ evaluation, and teachers’ self-evaluation. 
Before beginning the class design for lesson is evaluated by experienced teacher and then revised based on their 
feedback. Teacher rehearses the lesson process mentally and plans how to react to the expected learner’ responses. 
During the class, teacher can identify the learners’ learning status and organize the learning outcomes. After the 
class, teacher can ask the students who had a problem in learning about their difficulties. By writing the comments 
of experienced teachers and learners and learning outcomes, teacher reflects their teaching based on the expectation 
when he or she designs the instruction. 

The last step, ‘conduct summative evaluation’, is for confirming learners’ learning outcomes after finishing 
the unit. The test results are recorded and used to help students’ learning. 

The Review of the Initial Model by Experts and Results 

The initial model was reviewed by 6 experts and then revised based on their responds. Their opinions 
reflected on the second model are as followings. First, develop worksheets including various examples. Though 
specific guide and examples are suggested to increase the possibilities of applying the model to school, the need for 
worksheets are still asked to use model more easily. The comments of experts are mainly about activities of steps 
and examples in explanation report of the model. They suggest more activities and the need of developing various 
worksheets forms in order to choose the form according to teachers’ conditions. 

Second, revise the term. The most suggested opinion is the need for changing the terms. The terms used in 
the model are different to the teachers’, so it is difficult for teachers to understand the activity of the steps. Therefore, 
it is necessary to revise the term. 

Last, revise the linear process. The model suggested activities to follow in every step helping novice 
teachers use the model easily. At the same time, formative evaluation is applied to overcome the linear trait. 

However, each step is conducted simultaneously. Considering it, the model is revised to show the steps are 
overlapped. 

Though most of the participants respond positively to the model, they responded differently each other for 
the opinions that ‘analyze instruction resources’ is prior to ‘design instructional strategies’. However, considering 
the influence of abundant resources and positive aspects of the resources for novice teachers, the order is not revised. 
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The Development of the Second Model 



[Figure 2] visualized the second model. Activities of the step are same with the initial model but 
components of a unit and a lesson design are placed parallel to stress activities are conducted simultaneously. In this 
model, each step is not separate but incorporated and overlapped. 




Figure 2. The second instructional systems design model for elementary school teachers 



Also, the term is changed to help to understand the activities clearly. To show the purpose of the step 
definitely, ‘unit’, and ‘lesson’ is used repeatedly. In addition, ‘formative evaluation’ is altered to ‘evaluate the 
design’, since term of formative evaluation is used differently to elementary school teachers. 

The worksheets with specific guide and various forms are developed for teachers to use model (Figure 3). 




Figure 3. The example of worksheet 
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The experienced teachers’ designing methods are added to specific activities. In analyzing learning 
objectives and learning contents, reorganizing processes such as incorporation of lessons and changing the order of 
the lesson are highlighted. Also, the various evaluation methods including students’ self-evaluation and peer- 
evaluation are introduced. 

The Review of the Second Model 

The elementary school teachers, 3 novice and 3 expert, suggested the opinions about the second model after 
they designed the instruction by using the model. The expert teachers designed without any difficulty. They also said 
that the model is useful to plan the lesson effectively. Through observing their design process, it is identified that the 
components of the unit design are conducted at the same time and analyzing the resources is prior to designing 
instructional strategies. 

The novice teachers have some difficulties in designing instruction using the model. Especially, it is hard to 
control the scope of the unit design for them. They have a tendency to focus on the individual lesson rather than 
understanding the unit, so they design the unit including lesson design activities. They insist to provide more 
specific guide and examples and need various forms of worksheets to select according to their conditions. 

Commonly, they ask to simplify the things that they should write because they feel burden to use the model 
if there are a lot of things to do. Elementary school teachers must teach minimum 4 hours a day. Therefore, if there 
are many things they have to do when they design an instruction it is hard to use it. 

The results of the review revealed that the second model is effective to design instruction. However, more 
worksheets were developed and specific guides were added in the explanatory guidance. 

Conclusion and Discussion 

This research intends to guide an effective instructional design for novice elementary school teachers. The 
first model was developed based on the theories of instructional systems design model and principles of effective 
learning. The model was revised based on the resulting feedback of experts and elementary school teachers. 

Through this study, four characteristics of theoretical model to guide instructional design for teachers are 
suggested as follows. First, the first step of ‘analyze instructional resources’ are emphasized. ‘Design instructional 
strategies’ influences to develop instructional materials in previous instructional design models (Dick & Carey, 

1978; Dick, Carey, & Carey, 2005). The step of ‘analyze instructional resources’ is conducted before the step of 
‘develop instructional materials’ in elementary school conditions. It is possible because there are plenty of 
instructional materials in a various forms available. The activity of ‘analyze resources’ for effective learning 
positively influences on the instructional design. Especially it is useful for novice elementary school teachers who 
teach all subjects and have difficulties in designing instructional strategies for effective and efficient instruction. 

Second, it proposes a specific practice of formative evaluation for instructional design. The evaluation after 
finishing a class doesn’t help to improve the actual lesson (Lim & Yeon, 2006; Naumann & Jenkins, 1982). In this 
respect, formative evaluation steps are conducted after unit and lesson design. The comments of experienced 
teachers are helpful to reduce expected trial and error and effective instruction. 

Third, design lessons after design unit. The unit is composed of lessons. By designing lessons based on the 
design of a unit, former learning is systematically combined with the next. Experts and elementary school teachers 
in this study identified the effectiveness of this process. 

Forth, ‘analyze learning objectives’ is the first step of the model. The teachers usually focus on learning 
contents rather than learning objective when they plan the lesson (Clark & Peterson, 1986; Shavelson, 1983; Yinger, 
1980). It is because learning objective and contents are already fixed in national curriculum. Though a decision 
relating to analyze learners and learning environments effects another in the process of instructional design, the 
information about learners’ learning levels and learning environments are in teachers’ mental structures because 
elementary school teachers can observed students everyday in the classroom. Considering this condition, the first 
activity should be ‘analyze learning objective and instructional contents’ based on teachers’ existing information 
about learners and environments. 

Meanwhile, in this study, some implications of developing instructional systemic design model are 
identified by design-based research. First, in the process of designing instruction, teachers take steps concurrently. In 
the first model, steps are definitely distinguished to increase the understanding and possibility of using the model. 
However, reviewers in this study indicated that it may be impossible to divide activities of instructional design 
distinctly. In elementary school, it may be hard for teachers to spend much time planning the instruction, so steps of 
instructional design may have to be overlapped. 
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Second, it is necessary to use the terms to communicate with teachers. Every organization has its 
specialized terms. To develop the model for elementary school teachers, it needs to apply the term that elementary 
school teachers can understand and use. The ‘formative evaluation’ using in the initial model is transformed to 
‘evaluate the design’ since teachers use the ‘formative evaluation’ as the evaluation of students to check their 
learning. To communicate with the user of the model, it is important to select the terms used in the real contexts. 

Third, the job aids are necessary to support teachers’ instructional design. One of the reasons why teachers 
don’t design an instruction is that they have little time (Park, 2007). If it takes time to follow the steps of model for 
designing instruction, teachers will not try to apply it. Experts advised the needs of worksheets and the elementary 
school teachers also responded the importance of the worksheets in the interviews. Through the interviews with the 
teachers, the importance of providing worksheet with the model is identified to increase the possibility of using the 
model. 

In conclusion, through this study, the characteristics and design principles that can be applied to develop an 
effective instructional design model are identified. Although a variety of instructional design models have been 
developed by many researchers, teachers use the model rarely because of the theoretical limitation of the model and 
realistic problems (Park, 2007). In this situation, it is expected to use the model developed in this study widely since 
it is based on the understanding of real conditions. Also, for novice teachers, it is useful to plan the instruction 
effectively and efficiently. Furthermore, if it applies to pre-service teachers’ curriculum, it will help students design 
an instruction systemically. 

The further studies to investigate the improvements of the model by applying the model to a large number 
of teachers and schools and examine the effectiveness of the model are expected. 
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Abstract 

The purpose of this study is to investigate the design principles of e-Learning contents with IMS Learning Design 
(IMS LD). IMS LD is an open standard as framework and meta-language, and it is focused on learning activities. 
The design principles with IMS LD are important to have learning effects. However, the most studies related to IMS 
LD have focused on technical aspects such as standards, systems, authoring tools, or players. In addition, the studies 
did not explore the design principles for e-Learning content with IMS LD. 

We proposed an eight design principles of e-Learning contents with IMS LD, and we designed and developed and e- 
Learning contents which applied the design principles. The e-Learning content with IMS LD was about ‘needs 
analysis in educational technology’, and it was developed for supporting the environment of ‘Problem-Based 
Learning (PBL)’. The e-Learning content was evaluated by nine graduate students who were majored in educational 
technology to confirm the validity of design principles. 

Keywords: learning design, IMS Learning Design, design principles, e-Learning environments 

Introduction 

Educational paradigm is changing from instruction-focused paradigm to learning-focused paradigm. The 
learning-focused instructional design which reflects the learning-focused paradigm, the changes from 
standardization to customization, from a focus on presenting material to a focus on making sure that learners’ needs 
are met, from a focus on putting things into learners’ heads to a focus on helping learners understand what their 
heads are into, to instructional design (Reigeluth, 1999). Problem-Based Learning (PBL), Goal-Based Scenario 
(GBS) and so on are the examples of learning-focused instructional design. 

Learning design is the new concept of instructional design focused on learners, especially the sequence of 
learning activities that learners undertake to attain some learning objectives, including the resources and support 
mechanisms required to help learners to complete these activities (Koper & Bennett, 2008). It is similar to learning- 
focused instructional design. 

There are attempts to apply the idea of learning design to e-Learning. e-PBL, e-GBS, and simulation are the 
examples of learning design in e-Learning environments. These attempts that are learning design for e-Learning 
design support learning activities in e-Learning environments. But most e-Learning programs are based on specific 
theories or models on learning design. So there are many difficulties to support several e-Learning programs which 
have distinctive backgrounds. Therefore, there is the need for the framework that can integrate diverse theories and 
models. 

IMS Learning Design (IMS LD) is a good example to apply the concept of learning design for e-Learning 
and to support diverse theories and models. IMS LD is an application of a pedagogical model for a specific learning 
objective, target group and a specific context or knowledge domain (Koper & Olivier, 2004). In addition, IMS LD 
specification is an open standard as a framework and meta-language (IMS GLC, 2003a) that is used to design and 
develop a wide variety of e-Learning contents and systems in a formal, semantic, interoperable and machine 
readable way (Koper & Miao, 2009). IMS LD is focused on learning activities, including the resources and supports 
to help learners to complete those activities. It is based upon the premise that learning is different from content 
consumption, and learning happens when learners cooperate collectively to solve problems in social and work 
situations (Tattersall & Koper, 2003; Lukasiak et ah, 2005). In this point of view, IMS LD is quite distinct from 
traditional e-Learning standards such as Sharable Contents Object Reference Model (SCORM). 

The prior studies related to IMS LD could be classified into the following three themes: 1) Learning Design 
and ontologies, 2) developing Learning Design in terms of methods, patterns, and integration with other standards, 
and 3) Learning Design engines (Koper, 2006). In short, the main aspect of these studies with IMS LD focused on 
technical aspects such as standards, systems, authoring tools, and players, not the aspects of instructional design or 
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learning design. The other aspect of them was definitely about designing e-Learning contents or systems, but these 
have led to the development of products. Therefore, most of the studies related to learning design for IMS LD were 
about the development procedures of e-Learning contents. However, it is important to develop principles for 
designing e-Learning contents with IMS LD, because the learning effects come from designing models and ideas 
which can be applied to e-Learning contents and systems development, not the product itself. 

In this study, we proposed an eight design principles of e-Learning contents with IMS LD, and we 
developed the e-Learning content by those design principles. The e-Learning content with IMS LD was evaluated by 
graduate students who are majored in educational technology to confirm the validity of design principles. 

IMS Learning Design 

Overview 

IMS LD is the standard which is based on Educational Modeling Language (EML) in Open University of 
the Netherlands (OUNL). It provides a framework of elements that can be used to describe formally the design of 
any instruction-learning process (IMS GLC, 2003a). Because IMS LD is described as XML format, the instructional 
designers and learning designers who don’t know the language may feel uncomfortable. But the specification of 
IMS LD liberates from the use of non e-Learning specific or proprietary scripting languages to create learning 
processes (Olivier & Tattersall, 2005). 

IMS LD specification is the document which describes IMS LD, and supports design and development of 
e-Learning content and system based on IMS LD. It consists of three documents: IMS Learning Design Best 
Practice and Implementation Guide which represents design examples using IMS LD(IMS GLC, 2003a), IMS 
Learning Design Information Model which describes the elements and structures of IMS LD(IMS GLC, 2003b), and 
IMS Learning Design XML Binding which explains language rules(IMS GLC, 2003c). 

IMS LD specification consists of three levels: A, B, and C, where a learning designer and developer can 
choose the level. Level A is the basic, and level B consists of elements of level A, property, and condition. In 
addition, level C consists of elements of level B, and notification. Property contains the value of learning contents, 
learners, staffs, and so on. Property can be personalized by the results from conditions. Finally, notification means 
announcement on learning process. 

Elements 

Table 1 lists the major elements IMS LD. As Table 1 represents, IMS LD specification consists of title, 
learning-objectives, prerequisites, components, method, and metadata. The most prominent elements are 
components and method, the rest of them are optional. 

Table 1 . The major elements in hierarchical order adapted from IMS LD level A specification 

(superior) (subordinate) 

learning-design title 

learning-objectives 



prerequisites 



components 


roles 


learner 

staff 




activities 


learning-activity 

support-activity 

activity-structure 




environments 


environment 


method 


play 


act role-parts 



metadata 
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Components are the content elements of Learning Design which consist of roles, activities, and 
environments. Roles are learner and staff such as instructor, tutor, and manager. Activities are learning-activity and 
support-activity, and activity-structures. And environments support activities, and they are not only technical support 
also learning material. 

Method is the structural and procedural element of Learning Design. That is, method is related to learning 
process, and it has the metaphor of play. So the method consists of play, act which is part of play, and role-parts 
which connect act to roles and activities of components. 

Besides the elements of IMS LD level A in table 1, IMS LD has property, condition, and notification as the 
elements of IMS LD. Property and condition are the parts of level B, and notification is the part of level C. 

Conceptual Model 

Figure 1 is the model which represents the relations of IMS LD major elements. Table 1 is focused on 
technical aspects, but figure 1 is focused on conceptual aspects. So, it has not only IMS LD terms but general terms 
such as person, outcome. In addition, the model includes subordinate elements such as learning object, service. 



learning design 




Figure 1. The conceptual model of IMS LD level C 
Characteristics 

The characteristics of IMS LD are as follows. 

Learning designer designs based on learning activities of learner. Instructional design focuses on following 
pre-determined pathway, but IMS LD which reflects learning design considers more complex processes 
between teachers, learners, contents, and so forth (Sims, 2006). 

It can maximize the interaction. Because learning designer can design cooperative learning through roles in 
e-Learning environments, it can make increase the interaction between learner and learner. And it makes 
adaptive web-based learning possible through the interaction between learner and learning system. 

Design and development of e-Learning progress in pedagogical context. IMS LD consists of the terms and 
the contents related to learning. That is, design and development are based on the concepts of learning. So 
designing is highly correlated to development. 

Reusability extends. IMS LD can reuse not only learning objects, but learning activities, and structure. 
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Methods 



The design principles were developed by following two steps: 1) finding out design principles from 
reviewing the relevant literature, 2) modifying design principles by designing and developing e-Learning content 
with IMS LD. 

The e-Learning content with IMS LD was about ‘needs analysis in educational technology’, and it was 
developed for supporting the environment of ‘Problem-Based Learning (PBL)’. The reasons why we chose PBL 
were 1) PBL is effective learning model for improving learners’ active participation and it can foster learners’ 
intrinsic motivations and develop their self-learning skills (Barrows, 1985; Duffy & Cunningham, 1995; Savery & 
Duffy, 1995), 2) PBL is an appropriate learning model that can represent the characteristics of design principles with 
IMS LD. 

Workshop and interview for evaluating the design principles with IMS LD were held, and nine graduate 
students participated with the workshop or interview. They are majoring in educational technology, six of them have 
experienced designing or evaluating e-Learning contents, and two of the rest have experienced studying using e- 
Learning contents. That is, eight of nine participants have experienced e-Learning contents. The participants 
evaluated the design principles in e-Learning content with IMS LD as e-Learning designer, and their responses of 
the survey and interviews were analyzed to investigate the strengths, weaknesses, and improvements of the design 
principles. 



Results 



Eight Design Principles 

Reviewing the relevant literature, designing and developing e-Learning content with IMS LD led to 
identify eight design principles (Lim & Kim, in press). 

Design learning content and learning activity together. 

The existing e-Learning contents were focused mostly on learning contents, and learning activities were 
accomplished on another spaces or tools like bulletin board, chatting room, and so on. However, the e-Learning 
contents with IMS LD included both learning contents and learning activities. That is, designing e-Learning contents 
with IMS LD means not only designing learning contents but also designing learning activities. 

Design adaptive learning about learners’ characteristics. 

IMS LD is a standard for adaptive learning, so designing e-Learning contents with IMS LD should be 
focused on adaptive learning by analyzing contents and learners. Especially, IMS LD had standardized variables: 
property and condition. Therefore, the variables could be used in another e-Learning contents with IMS LD. Learning 
designer could set the learners’ characteristics as property, and design and present learning activity, learning contents, 
and control as condition. 

Design learning activity which are appropriate to roles. 

The existing e-Learning contents were based on individual learning, so they were not good at supporting 
collaborative learning or cooperative learning. But e-Learning contents with IMS LD were based on collaborative or 
cooperative learning as well as individual learning. In order to support collaborative or cooperative learning, IMS LD 
had the variable ‘roles’. Through roles, learning designer could set the roles of learner and staff that supported learner, 
and he/she could design the learning activity, learning contents, learners’ control, and function by roles. 

Set the range of each activity considering learning activity, content, and screen composition. 

When the existing e-Learning contents were designed, instructional designer designed considering topic and 
amount of learning contents. However, IMS LD was based on learning activity as well as learning contents, learning 
designer should design each learning activity with IMS LD considering learning activity, content, and screen 
composition. 

Design the activity which is devided between activity-description as main-activity and environment as sub-activity 
that support main-activity. 

Activity of IMS LD related to two main elements: activity-description and environment. Activity-description 
meant main-activity, and environment meant to support main-activity such as supplementary resources and supporting 
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tools. When learning designer designed e-Learning contents with IMS LD, he/she needed to devide main-activity and 
sub-activity. In addition, he/she should design activity-description as main-activity, and environment as sub-activity. 

Design the activity considering reusability or reproducibility. 

IMS LD was focused on learning-activity, and the elements including learning-activity were similar to 
learning object in terms of applying reusability or reproducibility. Specific guidelines from this design principle were 
as follows. 

Present the names of programs and lessons separated from activity-description and environment. 

Design the menus of activity-description and environment separately. 

Design activity-description and environment considering similarity and inclusion of their contents. 

Design the expected notification included. 

The existing e-Learning contents did not include notification, and notification was accomplished by 
instructor or tutor using tools such as Learning Management System (LMS) or Short Message Service (SMS) in 
implementing time. However, e-Learning contents with IMS LD could be designed to send the expected message 
automatically. Therefore, learning designer could design notification, and instructor or tutor only needed to notify the 
exceptional message using tools. In order to apply this design principle, IMS LD had the elements: property, time- 
limit, and notification. 

Design the timeline considering class type such as online-only class or blended class. 

The existing e-Learning contents were needed additional management. However, e-Learning contents with 
IMS LD could be designed to open and close the class automatically. Using the characteristic of IMS LD, blended 
class could be managed easily. Learning designer could make the e-Learning content opened when need automatically. 

Reponses of Respective Design Principles 

Most participants’ responded that each design principle was helpful for designing e-Learning contents with 
IMS LD. However, there were also some weaknesses and improvements for each design principle. Participants’ 
responses of each design principles were listed in Table 2. 

Table 2. Responses of respective design principles 



Design principles 


Participants’ responses 


strength 


weakness 


Improvement 


Design learning content 
and learning activity 
together. 


• It could expand the 
range of learning 
activity. 

• It enabled learner to 
learn intensively. 

• Helpful for increasing 
learning attitude 

• It enabled to design the 
e-Learning contents 
that were similar to 
practical instruction. 

• Helpful for studying 
easily 


• There was doubt 
whether it was 
effective. 

• It could limit learning 
activity as macroscopic 
concept. 

• It was difficult to learn 
only learning contents 
after class. 


• It is needed to provide 
guidelines to design 
learning activity. 
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Design principles 


Participants’ responses 


strength 


weakness 


Improvement 


Design adaptive learning 
about learners’ 
characteristics. 


• Good to apply learners’ 
characteristics 

• Helpful for providing 
success opportunity 


• Difficult to choose and 
gather learners’ 
characteristics 


• It is needed to clarify 
what learners’ 
characteristics would 
be chosen. 


Design learning activity 
which are appropriate to 
roles. 


• Helpful for designing 
collaborative learning 
or cooperative learning 

• It enabled to learners to 
interact and participate 
learning. 




• It should be 
emphasized to design 
and development the 
diverse learning 
models. 


Set the range of each 
activity considering 
learning activity, content, 
and screen composition. 


• It was learner-centered 
design. 

• Helpful for finding out 
the information about 
learning activity 

• Easy to understand 
what to do 


• It was difficult to apply 
the principle, because 
it would be different 
according to learning 
models. 


• It is needed specific 
guidelines. 


Design the activity which 
is devided between 
activity-description as 
main-activity and 
environment as sub- 
activity that support 
main-activity. 


• Helpful for paying 
attention the diverse 
tools as environment 


• There were some 
difficulties to 
understand the terms 
such as activity- 
description, 
environment. 




Design the activity 
considering reusability or 
reproducibility. 


• Helpful for developing 
e-Learning contents in 
terms of reusability 

• Effective at sharing 
good contents 


• It was difficult to 
develop using the 
existing contents. 

• It should precede to be 
standard. 




Design the expected 
notification included. 


• It could be immediate 
notification. 

• Helpful for managing 
learning schedule 

• Helpful for manager 

• Helpful for developer 


• It could be ineffective 
for learners, because 
automatical 
notification would be 
felt formally. 


• It is needed to provide 
not only e-mail, but 
also various device. 


Design the timeline 
considering class type 
such as online-only class 
or blended class. 


• It could reduce ‘false 
click’. 


• It could decrease 
learners’ leadership. 


• It is needed to clarify 
the terms of class. 
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Conclusion 



Learning design which is fundamental concept of IMS LD is focused on learner, especially the sequence of 
learning activities that learners undertake to attain some learning objectives, including the resources and support 
mechanisms required to help learners to complete these activities (Koper & Bennett, 2008). IMS LD is the main 
attempt to apply the concept of learning design for e-Learning, and to support diverse learning theories and models. 
The prior studies related to IMS LD had focused only on technical aspects, not the aspects of instructional design or 
learning design. In addition, most of the studies related to learning design with IMS LD were about the development 
procedures of e-Learning contents, not the design guidelines, principles, and models. 

This study aimed at investigating the design principles of e-Learning contents with IMS LD. The design 
principles were developed by following two steps: finding out design principles from reviewing the relevant literature, 
and modifying design principles by designing and developing e-Learning content with IMS LD. In addition, the 
strengths, weaknesses, and improvements of design principles were analyzed from the workshop and interview for 
evaluating the design principles. Through this study, major directions for improvements of the design principles were 
recommended, and an design-based research is suggested to examine the responses of learners for further study. 

Acknowledgement 

This research was funded by the Korea Ministry of Knowledge Economy (No. 10029965). 

References 

Barrows, H. S. (1985). How to design a problem-based curriculum for the preclinical years. New York: Springer 
Publishing Co. 

Duffy, T. M., & Cunningham, D. (1995). Constructivism: Implications for the design and delivery of instruction. In 
D. Jonassen (Ed.), Handbook of Research for Educational Telecommunications and technology’ (p. 170-198). 
New York: Macmillan. 

IMS GLC (2003a). IMS Learning Design Best Practice and Implementation Guide: Version 1. 0 Final Specification. 
IMS Global Learning Consortium, Inc. 

IMS GLC (2003b). IMS Learning Design Informational Model: Version 1.0 Final Specification. IMS Global 
Learning Consortium, Inc. 

IMS GLC (2003c). IMS Learning Design XML Binding: Version 1.0 Final Specification. IMS Global Learning 
Consortium, Inc. 

Koper, R. (2006). Current Research in Learning Design. Educational Technology & Society, 9(1), 13-22. 

Koper, R., & Bennett, S. (2008). Learning design: Concepts. In H. H. Adelsberger, Kinshuk, J. M. Pawloski, & D. G. 
Sampson (Eds.), Handbook on information technologies for education and training (2nd ed.) (pp. 135-154). 
Heidelberg: Springer- Verlag. 

Koper, R., & Miao, Y. (2009). Using the IMS LD Standard to Describe Learning Designs. In L. Lockyer, S. Bennet, 
S. Agostinho, & B. Harper (Eds.), Handbook of Research on Learning Design and Learning Objects: Issues, 
Applications and Technologies (pp. 41-86). Hershey, PA: IGI Global. 

Koper, R., & Olivier, B. (2004). Representing the learning design of units of learning. Educational Technology’ & 
Society, 7(3), 97 -111. 

Lim, C., & Kim, S. (in press). Design Principles and Procedures for E-learning Programs with the IMS Learning 
Design Specification in Higher Education in Korea. In T. Islam, O. Lee, J. Peterso, & M. Piscioneri (Eds.), 
Effectively Implementing Information Communication Technology’ in Higher Education in the Asia-Pacific 
Region. Hauppauge, NY: Nova Publishers Inc. 

Lukasiak, J., Agostinho, S., Bennett, S., Harper, B., Lockyer, L., & Powley, B. (2005). Learning objects and learning 
designs: an integrated system for reusable, adaptive and shareable learning content. ALT-J, Research in 
Learning Technology’, 13(2), 151-169. 

Olivier, B., & Tattersall, C. (2005). The Learning Design Specification. In R. Koper, & C. Tattersall (Eds.), Learning 
Design: A Handbook on Modelling and Delivering Networked Education and Training (pp. 2 1 -40). Berlin- 
Heidelberg: Springer Verlag. 

Reigeluth, C. M. (1999). What Is Instructional-Design Theory and How Is It Changing? In C. M. Reigeluth (Ed.), 
Instructional Design Theories and Models Volume II: A New Paradigm of Instructional Theory (pp. 5-47). NJ: 
Lawrence Erlbaum Associates, Inc. 



170 




Savery, F., & Duffy, T. (1995). Problem-based learning: An instructional model and its constructivist framework. 
Educational Technology’, 35(5), 31-38. 

Sims, R. (2006). Beyond Instructional Design: Making Learning Design a Reality. Journal of Learning Design, 7(2), 
1-7. 

Tattersall, C., & Koper, R. (2003). EML and IMS Learning Design: from LO to LA. Retrieved March 31, 2010, from 
http://dspace.ou.nl/handle/1820/107. 



171 




Exploring the Educational Use of an Augmented Reality Books 



Cheolil Lim, Taejung Park 
Seoul National University, Republic of Korea 



Abstract 

This study aims at exploring and promoting the availability of AR books to augment real and physical 
books with virtual content for education. Above all, previously developed cases as well as literature review relating 
to AR books provide an introduction to what an AR book is and its educational effects and technical characteristics. 
To achieve this goal, we analyzed 13 research cases of developed AR books according to criteria of both educational 
and technical uses and examined needs of 42 education professionals via an online survey. Based on case analysis, 
AR books increase the educational effects on affordance, reading comprehension, memory, concentration, 
interactivity, problem solving, and imagination. It is found that they also enable students to engage in self-paced and 
level-differentiated learning. In terms of technical uses, AR books utilize mainly a tangible user interface while 
having various displays and markers owing to their unique technical features. Based on needs analysis, education 
professionals express the willingness to utilize AR books for educational purposes in spite of their low awareness of 
them. So in a nutshell, the results of the case and needs analyses suggest the potential of AR books to be applied to 
and used in educational environments. Therefore, successive studies on the instructional design and development of 
AR books along with empirical research on their multilateral learning effects are required. 



Introduction 

The rapid development of new digital devices and technology has offered both innovative opportunities and 
challenges for instructional designers and teachers. Among the newer types of digital media, Augmented Reality 
(AR) will become common. This type has been applied to numerous disciplines, including medicine, the military, 
engineering maintenance and repair, and entertainment. AR is an emerging technology tool that could afford a 
variety of learning opportunities to expand new interactive constructivist learning environments. Research in the 
field of AR is essentially technology-driven, which means that the needs and effectiveness of the educational 
applications remain undiscovered and a minor concern. Over the last two decades there have been numerous efforts 
and much prediction regarding the replacement of real books with digital equivalents such as electronic books 
(Grasset, Dunser, & Billinghurst, 2008). Along with the development of a variety of software content, the rapid 
growth of hardware such as e-Book readers, tablet PCs, netBooks, electronic ink (e-Link), and electronic paper (e- 
Paper) transcend the limitations of paper books, which have the difficulty of updating the information with 
knowledge-centered, and one-sided, linear learning materials. However, not everyone welcomes the coming loss of 
paper-based books, which have the strengths of transportability, flexibility and robustness (Back, Cohen, Gold, 
Harrison, & Minneman, 2001; Marshall, 2005), as an electronic book is clearly not equivalent to a real book with 
real paper pages. 

With the help of the release of augmented reality books which combine digital content with physical 
analogue books, we don’t have to make an effort to choose e-Books or paper based books. Recently there are 
increasing numbers of cases of the research development of AR books by Korean and international computer 
science labs. However, the researchers tend to focus on the technical aspects of AR books, not educational ones. 
Thanks to the rapid development of information and communication equipment, not only handheld device, head 
mounted devices, and PCs but also smartphones include the display, marker, and camera features required for AR 
books. Therefore, it is becoming easier for AR books to be used for educational purposes. The aim of this study is to 
explore the potential effects of AR books in education. To achieve this goal, we examined works related to reviews 
on technological features and educational use and conducted a needs analysis to assess educators’ needs. A 
collective review of the study results, including a literature review and needs analysis, indicated that AR books have 
positive learning effects on both cognitive and affective domains, demonstrating their potential for use in classroom 
environments. 
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Theoretical Background 



The evolution of e-Books 

The concept of e-Books is changing in accordance with the trends of the times and technological 
development. The first-generation e-Books are digitalized books; they are characterized by the formatting of texts 
and images into xml or pdf files while retaining the original content and forms of paper books. However, they have 
limitations in offering various layouts and multimedia and an interactive reading environment. The second- 
generation e-Books are multimedia books that are a combination of a variety of texts, pictures, sounds, 2D 
animation(gif), and video files (flash), thus providing interactive multimedia materials and interactive environments 
to readers. However, they have some difficulty in increasing learners’ immersion and presence and, furthermore, 
predicting active interaction between learners and multimedia materials (Lee, & Cho, 2002). Finally, the third- 
generation e-Books are AR books utilizing context-aware AR technology being actively researched and developed 
at present. They mix multimedia materials which allow learners to experience the multiple senses of touch, smell, 
and taste as well as sight and hearing by applying AR technology, thus offering additional information which paper 
books cannot and enhancing learners’ interaction(Ha, Lee, & Woo, 2009). These AR books are expected to enhance 
the strengths of and compensate for the weaknesses of paper books and other e-Books by adding diverse digital 
content to physical paper books with AR technology. Table 1 shows the development stages of e-Books. 



1 st generation e-Books 




Table 1. The Development Stages of e-Books 

2 nd generation e-Books 3 rd generation e-Books 





Digitalized books 

characterized by the 
conversion of the content 
and form of paper books into 
xml or pdf file format 



Multimedia books offering 
various multimedia and 
interactive environments to 
readers due to a combination 
texts, pictures, sounds, 2D 
animation (gif), and video 
files (flash) 



AR books providing virtual 
contents which offer contexts 
to paper-based books with a 
multi-sensory (visual, 

auditory, tactile, olfactory, 
gustatory) experience that 
support readers’ interactions 
in immersive environments 



Augmented Reality(AR) books 
- Concepts and features 

The concept of AR can be easily defined through the mixed reality continuum (see Figure 1). AR is mixed 
reality that is closer to the real environment. AR allows users more immersion and presence by seamlessly merging 
the virtual environment and the real environment (Azuma, 1997). 



| Mixed Reality (MR) 1 

◄ ► 

Real Augmented Augmented Virtual 

Environment Reality (AR) Virtuality (AV) Environment 

Figure 1. Mixed Reality Continuum (Milgram & Kishino, 1994) 

AR offers a realistic experience due to its multi-sensory 3D models and transitional interface which allow 
users seamless interaction between the real and virtual worlds and objects (Billinghurst, 2002). AR is simply the 
conveyance of data from the virtual world to the real world beyond the form of annotation data to generate a new 
world of enhanced visualization (Lee, & Ham, 2010). In addition, AR is a context-aware computer technique which 
can enhance interaction, providing users with the necessary information in real time by detecting their location and 
situation in a context (Azuma, 1997). 

With the beginning of the twenty-first century, the emergence of PDAs, camera phones, and wireless the 
internet induced the innovative growth of AR technology. Since 2008, with the development of as AR application 
for mobile phones, interest in AR has dramatically increased. Accordingly, AR books have appeared as an attempt 
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to apply AR to the e-Book format. AR books have enhanced the concept of books by creating interactive 
environment that incorporates animation, 3D graphics, and simulation (Shelton, 2002). AR books also have 
augmented physical paper books with 3D objects, video, voice, and multimedia elements generated by computer 
graphics using AR technology (Ryu, Cho, & Yang, 2009). AR books not only increase overall realism but also 
provide familiarity to readers so that they may interact in an environment based on the real physical environment 
(Park, 2009). Moreover, AR books are intriguing, thus prompting readers’ motivation and immersion (Korea 
Education & Research Information Service, 2006). A wide variety of AR books ranges from those which enable 
readers to simply observe the augmented 3D objects to those which require readers to actively interact with the 
books (Park, 2009). AR books are referred to by different names such as augmented book, AR multimedia books, 
digilog books, AR mediated books, 3D books, virtual pop-up books, and MagicBook. 

- Effects of educational use 

Shelton(2003) suggests that AR facilitates active, constructive, intentional, practical, and collaborative 
learning. Also, AR-based learning content reduces the acquisition of wrong concepts and raises the understanding of 
complex content (Shelton & Hedley, 2002). Therefore, AR leads to users’ curiosity and interest because users can 
easily observe and manipulate learning content (Korea Education & Research Information Service, 2006). In 
addition, several other studies (Dunleavy, Dede, & Mitchell, 2009; KERIS, 2006; KERIS, 2005; McKenzie, & 
Darnell, 2003; Noh, Ji, & Lim., 2010; Yeo, 2009) have reported the educational benefits of AR content on learning 
achievement and attitudes by discussing concepts such as immersion, satisfaction, presence, creativity, curiosity, 
motivation, engagement, and interest. 

Like AR-based learning content, AR books can support immersive learning, improve the understanding of 
complex information and increase motivation and participation (McKenzies & Darnell, 2003). This type of 
electronic book enables complex and various user interactions through the physical use of a real book. Using a real 
physical book for the development of an AR book has proven to have a positive influence on improving users’ 
enjoyment, engagement and usability (Grasset et ah, 2008) and reducing their cognitive load (Neumann, & Majoros, 
1998). Using AR books in educational settings leads to active participation, authentic learning and collaborative 
learning (Billinghurst, Kato, & Poupyrev, 2001; Shelton, 2003) and better understanding (Chen, 2006; Klopfer, Yoon, 
&Rivas, 2004; Shelton, & Hedley, 2002). Moreover, AR books can support teaching and learning by offering 
students the possibility of learning-by-doing (Doswell, Blake, Green, Mallory, & Griffin, 2006; Fjeld, & Voegtli, 
2002), facilitating students' information searches as information can be provided when and where needed 
(Cooperstock, 2001), and reducing the likelihood of errors due to the availability of necessary data and cues with 
little effort (Neumann, & Majoros, 1998; Regenbrecht, Lum, Kohler, Ott, Wagner, Wilke, & Mueller, 2004). 

Besides the benefits of learning achievement and attitudes, AR books have positive influence on varied and 
dynamic interaction. According to Azuma (1997), AR book systems have the following three characteristics: 1) they 
combine the real and the virtual, 2) they are interactive in real time, and 3) they can be registered in 3D. As the user 
freely rotates, turns, touches or takes a closer look at a book, the AR book system reacts accordingly (Saso, Iguchi, 

& Inakage, 2003). AR books are expected to support active interaction as well as learning and collaboration through 
the user interface(0 'Malley & Fraser, 2005; Price, et al., 2003) and facilitate intuitive interaction since the real 
context: real paper books and the real-world environment remain (Grasset, et al., 2007). Additionally, using AR 
books students can reorganize a story by closely interacting with specific components of interaction tools and 
creatively writing and editing its parts(Dunser & Hornecker, 2007). In short, AR books have a significant positive 
impact on achievement, attitude and interactions which can drive self-directed learning 

- Technological components of AR books 

In the case of the PC monitor as a display, the operating principles of the AR book learning system are as 
follows. First, the viewer program of the AR book is installed on the PC. Second, the PC recognizes and tracks the 
book’s pages through the input image from the camera to the computer. Third, 3D models, video, sound, and other 
multimedia content stored on the computer are registered and augmented to the book. Even though the book moves, 
augmented content is still rendered on the book. Fourth, users can interact with augmented virtual content on the 
book by turning over book pages or manipulating physical objects as a user interface and experience responses of 
the virtual content with multiple senses (Ha et al., 2009). Figure 2 shows the AR book learning system with a PC- 
based monitor as one of the displays. 
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web Camera 




Figure 2. AR book learning system 

The AR book system can be overviewed through essential technological components such as the displays, 
tracking methods and user interfaces, all of which have implications on educational settings. 



• Display 

The display technology for AR books are divided into Head Mounted Display(HMD) and Non-Head 
Mounted Display(Non-HMD) as shown in Table 2. 



HHD 



Table 2, Display devices 

Non-HHD 



HHD 



Spatial Display 



PC Monitor 
Display 





• Tracking 

Two different types of tracking technology exist: marker-based and marker-less. Using markers, the AR 
book system tracks and recognizes images for virtual objects. In early AR development studies, visible markers 
were used to easily find camera images. However, visible markers have the problem of decreasing students’ 
immersion in AR books. Therefore, marker-less tracking technology is being studied, which uses not visible markers 
but existing information, is being studied (Lowe, 2004). 



Marker-based Tracking Marker-less Tracking 

(Park, 2009) (Scherrer, Pilet, Fua, & Lepetit, 2008) 




Figure 3. Marker-Based and Marker-Less Tracking 



• User Interface 



175 







AR books are created by combining a traditional user interface, a Graphical User Interface (GUI) consisting 
of a mouse, keyboard, and screen; and a Tangible User Interface (TUI) consisting of physical objects such as 
marker-embedded paddles, cubes, and real-life objects (Brave, Ishii, & Dahley, 1998). The TUI allows students to 
easily manipulate content by touching, selecting and moving real objects in a more natural environment (Ishii & 
Ullmer, 1997). This TUI compared to GUI enables more immersive and intuitive interaction among multiple users in 
collaborative environments as shown in Table 3. 

Table 3. GUI-based and TUI-based Individual & Collaborative Forms of Interaction (Brave, Ishii, & Dahley, 1998) 
Individual Learning Environment Collaborative Learning Environment. 




Case Analysis of AR Books 



Method 

For the selection process of developed AR book cases, we searched for related research papers using 
keywords such as AR (AR-based) book, augmented book, AR electronic book (e-book), mixed reality (MR) book 
and other such terms on Korean and international databases from March of 2010 to July of 2010. Thirteen cases 
were chosen as shown in Table 4. The case analysis criteria were divided into educational use (e.g., related subjects, 
content, effects, target and method of use) and technological characteristics (e.g., displays, tracking methods and 
user interfaces). 



a 



b 



c 



d 



Table 4. Selected AR books for Case Analysis 



Name 



Screen Shot 



MagicBook 



Little Red 



AR 

Volcano 



Vivid 

Encycloped 

ia 




Researcher(s) or Developer(s) (Development Year) 
Source 

Billinghurt, Kato, & Poupyrev (2001) 

Retrieved from 

http://citeseerx.ist.psu.edu/viewdoc/download7doFl0. 
1.1.100.21 94&rep=rep 1 &type=pdf 



Saso, & Inakage (2003) 

Retrieved from 

http://portal.acm. org/ftgatcwav.cfm?id=965573&typc= 
pdf 



Woods, & Billinghurst (2004) 

Retrieved from 

http://www.hitlabnz.org/wiki/ARVolcano 



Shibata et al. (2004) 

Retrieved from 

http://www.ritsumei.ac.ip/~asa/publication/VR2004- 

T16.pdf 
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Augmented 
Story Book 


w 


s 

• Dunser, & Hornecker (2007) 

• Retrieved from http://www.ehornecker.de/Papers/GI- 

si AR-fmal2.ndf 


Virtual 
f Pop-Up 

Book 


mm 


• Taketa, Hayash, Kato, & Noshida (2007) 

• Retrieved from 

httn://Dortal.acm.org/citation.cfm?id= 1 76659 1 . 1 76664 

^ 5 


Aliens & 
g UFOs 


9 , * rwi 
I/MAS2 


• Metaio Co. (2008) 

• Retrieved from 

http://www.wired.com/gadgetlab/2008/10/im-in-yur- 

1 phvsi/ 


The House 
h That Jack 

Built 


|S! 


• Grasset, Dunser, & Billinghurst (2008) 

• Retrieved from 

http://ir.canterburv.ac.nz/bitstream/10092/239 1/1/1 261 

9 32362008- 

EdutainmentMixedRealitvBookVisuallvAugmentedlllu 

strativeChildrensBook 


The 

i Haunted 

Book 


IL 


• Scherrer, Pilet, Fua, & Lepetit (2008) 

• Retrieved from 

http://cvlab.epfl.ch/puhlications/publications/2008/Sch 

errerPFL08.pdf 


j wizQubes 


ps 


• Zhou, Cheok, &Tedjokusumo (2008) 

• Retrieved from http://www.iivr.org/issues/issue4- 
2008/9- 1 6%20wIzOubes-SZ-V3 .pdf 


Jack the 

k Time 

Traveller 


! s *SPI 


• Looser, & Clark (2009) 

• Retrieved from 

http://www.hitlabnz.org/wiki/3DMagicbookdisplavatU 

Clibrary 


. English 

Textbook 


: 4 * 

ooocrf^ 


• Cho et al. (2008) 

• Retrieved from 

http://citeseerx.ist.psu.edu/viewdoc/download7doU10. 
1 . 1 .99.4842&rep=repl&tvpe=pdf 


Digilog 

Book 


3D ^ ^>*1 USB 

In i M 

Book Vcnl 


• Ha, Lee, & Woo (2008) 

• Retrieved from http://cti.gist.ac.kr/ 



Results 

- Analysis of educational use 

Most research and development practices related to AR books were carried out with a technological 
approach centering on the field of computer engineering. Therefore, we had difficulty in finding cases which discuss 
AR books in terms of instructional design, teaching and learning theory and models and other educational aspects. 
Nevertheless, this study tried to analyze the selected cases according to related subjects, content, educational effects, 
and target users as follows. According to the results of the educational analysis, AR books are able to facilitate 
students’ comprehension, memory, concentration, affordance, interactivity, imagination, problem-solving and level- 
differentiated learning by offering them immersion, presence, and context. 

Table 5. Results of Educational Analysis 
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Subjects 


Content 




Educational Effects 


Target 


a 


Literature 


Reading stories 




improving immersion, imagination, 
reading comprehension 


K-l-6 


b 


Literature 


Reading and creating stories 




Increasing interactivity 


K-l-6 


c 


Science 


Experiencing volcano eruptions 




Increasing understanding of 

principles 

Driving self-paced learning 


K-7-12 


d 


Science 


Multifaceted and detailed 
observation and manipulation 
of 3D insects (moving and 
resizing) 




Increasing understanding and 

interactivity 


K-l-12 


e 


Literature 


Reading stories and completing 
activities 




Improving understanding with 

scaffolding 

Increasing motivation and interest 
Improving concentration, memory, 
affordance, and problem-solving 


K-l-6 


f 


Literature 


Reading stories 




Improving understanding 


K-l-6 


g 


Science 


Experiencing the planets and 
spacecrafts 




Increasing immersion 


K-7-12 


h 


Literature 


Reading the storybook and 
revising stories 




Obtaining positive feedbacks on 
interest, usability, and participation 


K-l-6 


i 


Literature 


Describing the content of 
poems with pictures 




Increasing understanding and 

immersion 


K-l-12 


j 


Literature 


Reading the storybook and 
creating stories 




Improving interactivity 


Babies & 
toddlers 


k 


Career Course 


Comic book about a journey 
through the history of cinema 
to find an interesting job 




Increasing immersion 


K-7-12 


1 


Foreign 

Language 


Comparing words of similar 
pronunciation for the study of 
vocabulary in an English 

textbook 




Increasing motivation and academic 
achievement 


K-4&5 


m 


Social Studies 


Observing and experiencing 
various Korean Buddhist 

temple bells through direct 
interaction 




Improving understanding 


K-l-12 



- Analysis of technological characteristics 

The following Table 6 shows the results of an analysis of the technological characteristics of thirteen 
selected AR books. 



Table 6. Results of Technological Analysis 



Items Types 


a 


b 


c 


d 


e 


f 


g 


h 


i 


j 


k 


1 


m 


HHD 


V 














V 






V 






Display HMD 




V 


V 


V 




V 
















PC monitor 










V 


V 


V 


V 


V 


V 




V 




Embedded in physical 
Marker- objects 




V 






V 






V 




V 




V 


V 


Tracking based 

Embedded in the book 


V 


V 


V 






V 
















Marker-less 








V 






V 


V 


V 




V 




V 


User GUI 










V 


















Interface -puj 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 


V 
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Needs Analysis of AR Books 



Method 



Based on the analysis results of the thirteen developed AR book cases, we developed questionnaires to determine 
the needs of education professionals for their use of AR books in educational settings (i.e., appropriate subjects, 
content, application methods, teaching and learning methods, content types for using them, criteria to determine 
whether or not to use them, and requirements for adopting their use in schools) as well as levels of awareness of AR 
and AR books and the perception of their educational effects (Table 7). In order to explore the possibility of the 
educational use of AR books, thirty-one elementary and junior high school Korean teachers and eleven researchers 
of Korean organizations related to elementary and secondary education informatization participated in an online 
survey conducted from August 4, 2010 through September 3, 2010. 




Table 7. Survey Items 


Survey Area 


Detailed Items 


Awareness of AR books & 
perception of their 

educational effects 


• Awareness of AR books 

• Perception of AR books’ educational effects 

• Willingness to use AR books 


Needs of educational uses of 
AR books 


• Appropriate subjects for AR books 

• Appropriate content types, application methods, teaching and 
learning methods for AR books 

• Criteria to determine whether or not to use AR books 

• requirements for adopting the use of AR books in schools 



Results 

The results of the survey on awareness, perception, and needs concerning AR books are shown in the 
following Table 8. Consequently, a large number of survey participants predict the positive effects of AR books on 
educational use and would like to use AR books for educational purposes. 



Table 8. Results of Needs Analysis Response No. (Response Rate %) 



1) 


Basic knowledge of AR books 




2) 


Whether to experience 


AR books 


a) 


never heard of 


31(74%) 


a) 


no experience 


39(93%) 


b) 


have heard of 


11(26%) 


b) 


plan to use 


3(7%) 



3) Appropriate subjects for AR books (multiple responses) 



a) 


science 


21(23%) 


• 

• 

• 


Observing animal and plant life 
Simulation of volcanoes and earthquakes 
Presenting 3D universal models 


b) 


art & physical education 


17(19%) 


• 

• 

• 


Presenting exercise postures 

Observing the engraving process 

Offering video clips on how to play musical 

instruments 


c) 


social studies 


13(14%) 


• 

• 

• 

• 


Presenting history- and geography- related 
materials 

Viewing geographical maps 

Visiting and experiencing historic sites, 

museums, and foreign countries 

Offering supplementary materials about 

environment and culture 


d) 


literature 


11(12%) 


• 

• 


Using as motivation cues in the introduction of 
a story 

Presenting related materials to literature 


e) 


foreign language 


11(12%) 


• 

• 


Studying listening and speaking through 
authentic conversation 

Offering situations and contexts for 
conversation 
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Observing how native speakers pronounce 



f) math 



g) technical education 



10 ( 11 %) 

8(9%) 



Representing geometrical figures in a 3D space 

Showing methods and procedures of crafts and 
cooking 



4) Appropriate methods for taking advantage of AR 
books (multiple responses) 


5) Suitable development forms of AR books for 
regular classes 


a) regular curriculum 


21(31%) 


a) supplementary materials 


35(83%) 


b) extracurricular activities 


12(18%) 


b) textbooks 


7(17%) 


c) self-study 


11(16%) 






d) alternative courses 


11(16%) 






e) level-differentiated 
learning 


7(11%) 






f) after-school classes 


5(8%) 






6) Appropriate teaching-learning methods for the 
use of AR books (multiple responses) 


7) Appropriate content types of AR books (multiple 
responses) 


a) discovery learning 


14(31%) 


a) simulation 


21(38%) 


b) direct instruction 


10(23%) 


b) presentation of materials 


11(19%) 


c) coaching 


9(20%) 


c) game 


9(16%) 


d) individual learning 


6(13%) 


d) storytelling 


7(13%) 


e) collaborative learning 


6(13%) 


e) drill and practice 


4(7%) 






f) tutorial 


2(4%) 






g) problem-solving 


2(4%) 


8) Evaluation criteria of AR books (multiple 
responses) 


9) Required supports to adopt AR books (multiple 
responses) 


a) quality of content 


14(31%) 


a) content 


17(35%) 


b) interaction 


13(29%) 


b) manual 


11(23%) 


c) user-friendliness 


1 1(25%) 


c) teaching and learning 
model 


6(13%) 


d) technology 


6(13%) 


d) lesson plan 


6(13%) 


e) price 


1(2%) 


e) casebooks 


5(10%) 






f) video clips of exemplary 
lessons 


2(4%) 






g) worksheet 


1(2%) 


10) Expected educational impacts of AR books 


11) Intent to educationally use AR books 


a) very effective 


5(12%) 


a) will use 


35(83%) 


b) effective 


23(55%) 


b) will not use 


7 (17%) 


c) uncertain 


14(33%) 





Discussion & Conclusions 

This study, through case analyses of AR books, shows that they have educationally positive effects on not 
only cognitive but also affective domains such as engagement, presence, interactivity and affordance. Additionally, 
the results of a needs analysis suggest the potential of AR books for application and use in classroom environments 
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since the willingness and intent of education professionals to use AR books for teaching and learning has been 
proven. However, efforts to seek out ways to use AR books educationally with a consideration of their technological 
features should be made. Additionally, empirical research on their multilateral learning effects should be conducted, 
and usability evaluations of these types of books should be carried out by adopting them soon. Specifically, while 
systemically establishing teaching and learning theories and methods, instructional design principles and the models, 
strategies, guidelines/manuals, casebooks, lesson plans, and worksheets of AR books, they should be gradually 
spread throughout schools. AR books, through the use of AR technology, have the potential to introduce innovation 
into education and open up new horizons in learning. 

AR book technology is entering the practical application stage, and currently, several AR books have been 
developed as a mobile cell phone application. Cell phones with GPS (a Global Positioning System) can be viewed as 
a place-independent AR system which can be superimposed onto any physical area. Using these cell phones to 
display virtual content in a real environment, a curriculum could be implemented immediately apart from the school 
building, in an area such as a school playground, a sports field, or a house. In the near future, the most likely 
platform for AR-based instructional models will shift to GPS-enabled cell phones such as the iPhone (Dunleavy et 
al., 2009). Although AR book technology is not new, its potential in education is just beginning to be explored. The 
utilization and contribution of AR books takes into consideration both technological and educational factors. The 
technological characteristics of AR books might make them useful instructional media for realizing a U-learning 
environment. Consequently, AR books should be designed and implemented in an effective and efficient manner for 
both teachers and students. Therefore, educators inclusive of researchers, instructional designers, teachers and other 
professionals should work with researchers and developers of AR book technology to find out how these 
technological characteristics can be optimally applied in an educational environment. Designing an effective and 
efficient learning environment using new technology such as AR is challenging. Future successive work ought to 
involve improvements for enhancing teaching and learning with little effort and at a low cost. 
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Abstract 

With the advent of Web 2.0 and "user-friendly" technologies, students have a vast array of e-communication tools at 
their fingertips. This session presents the results of a survey of 316 college students about their perceptions of 
appropriate technology use. Generational, ethnic, and gender issues including students' e-relationships with peers 
and social networking sites are explored. Significant generational issues, the need to implement cell phones in 
courses and pragmatic rules/etiquette (do's and don'ts for cell phone use) are suggested. 

Introduction 

With the advent of new and emerging "user-friendly" technologies and Web 2.0 tools, college students 
have a vast array of asynchronous and synchronous communication tools at their fingertips. Students can easily 
email, text, and instant message each other on bulletin boards, blogs, and social networking sites such as Facebook 
and Myspace. 

Flowever, as new technologies play an ever increasing role in our culture and the global community at 
large, it is not always clear what is considered to be appropriate and polite behavior (Rosenfeld & O'Connor- 
Petraso, 2010) as boundaries between public and private spaces (Wei & Leung, 1999) become obscured. For 
example, some political pundits believe Barak Obama secured his presidency through his social networking site, 
mybarakobama.com, (Organizing for America, 2007), akin to individuals and businesses, who both nurture and 
expand their relationships and associations through Web 2.0 tools. 

Although there have been several attempts to draft guidelines (Elgan, 2010; Krotz, 2010; McCluggage, 
2009) for texting or making or taking a cell call (such as using a lowered voice and/or avoiding personal topics), and 
one out of every two Americans polled almost a decade ago believed cell phone users were generally discourteous 
(Srivastava, 2006), there are still no commonly accepted standards or Robert's Rules of "Appropriate Use of 
Cell/Smart Phones." What we do know from present research (Subrahmanyan & Greenfield, 2008; Medina, 2010) 
is that schools in general are trying to control for distracting uses of the Web and cell/smart phones. 

As reported by Rosenfeld and O'Connor-Petmso (2010, p. 263): 

Many schools (including New York City public schools) ban the use 
of cell phones because they are found to be disruptive and promote 
cheating. Flowever, since the tragedies of September 11 and Columbine, 
school districts are reconsidering their usefulness since students bring 
their cell phones to school so that they can stay in contact with their 
parents. 

The present research builds upon work by the authors who previously surveyed 129 college students’ 
perceptions of appropriate and inappropriate technology use in the classroom and public spaces. Although the 
authors did not find any significant correlations among gender, culture, ethnicity and cell/smart phone use in the 
classroom and public places, a "generational" theme did emerge. They found that the younger age group (20-29 
years old) overwhelmingly supported texting in class and public places "when bored" and they used cell phones in 
an "emergency" and "depending upon whom you are with" situations. 

In the prospect of replicating or disputing their prior research, the authors refined their survey to include 
both their "frequency of texting and/or cell phone use on a daily and weekly basis" and their "perception of 
appropriate and inappropriate technology use" in both the classroom (with the professor, collaborating with 
colleagues, during a lecture, conference/meetings, etc.) and public places such as doctor's offices, museums, 
libraries, and restaurants. In addition, the new survey asked participants to gauge both their involvement and 
perception of their level of expertise in e-communications (such as Skype and social networking sites), their 
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perception of their professor's skill level in technology, and their desire to see new and emerging technologies 
implemented into their college coursework. The present research surveyed 316 college level students and is part of 
an ongoing international research study to ascertain an individual’s perceptions of appropriate use of technology. 

Based upon the findings of the aforementioned survey research, the authors will present a few pragmatic 
guidelines (do's and don'ts) for appropriate use of cell/smart phones in both the classroom and public places. It is 
their hope that these guidelines will make "sense" to the students who in turn will then adopt and practice 
appropriate use of technology. 



Methods 



Participants 

The present survey consists of 316 undergraduate (58%; n=184) and graduate (42%; n=132) level students 
from an urban university in the northeast and several universities in China. Approximately three-fourths of the 
participants are female and single (n=235). Sixty-nine percent (n=217) of the participants range in age from 20-29 
years old, followed by the 30-39 year-old age group (14%; n=45). The remaining age groups, 13-19 years old, 40-49 
years old, 50-59 years old, and 60+ respectively, range in size from 8% (n= 24) to .6% (n=2). 

Sixty-nine percent (n=218) of the participants are from the American university and thirty-one percent of 
the participants (n=98) are from universities in China. Although all the Chinese participants were born in China, the 
American students have diversified birth origins; the largest number of these participants, 51% (n=162) were born in 
the United States, followed by 18% (n=57) who were born in either the Caribbean, Europe, Central America, and/or 
Pakistan. The four largest ethnic groups (87%; n=274) represented in the study are Chinese (33%; n=104), 

Caucasian (29%; n=91), Black/Non-Hispanic (16%; n=50,) and Hispanic/Latino (9%; n=29). 

Instrumentation 

The self-administered questionnaire used for the analysis is the International Technology Etiquette Survey, 
v. 3 (O'Connor-Petruso, Rosenfeld, Martinez-Pons, 2011). The survey instrument was pilot tested and has a 
Cronbach's alpha reliability coefficient of .791. The survey consists of 60 questions and is divided into three parts: 
Part I) Demographics & Background Information, Part II) Frequencies, and Part III) Agree/Disagree (set to a Likert 
Scale from 1: Strongly Disagree, 2: Disagree, 3: Agree, and 4: Strongly Agree). 

Procedure 

All participants were asked to take the online International Technology Etiquette Survey (located at 
http://internationaltechnologystudies.org) and select either the English version or the Chinese version. Data were 
collected over a five-month period. 



Analysis 

The data were analyzed using IBM's PASW (Predictive Analytics Software), V. 19. Descriptive statistics, 
correlations, and t-tests were run to ascertain frequencies, linkages, and mean differences. Significant results are 
reported. 



Results 



Frequencies 

One-hundred percent of the participants report they use either a cell phone (39%; n=124), smart phone 
(45%; n=166), or both (7%; n=26). Approximately 67% ( n=21 1) of the participants report they use their cell/smart 
phone to make/take calls on average between 1-6 times per day. Fifty-three percent (n= 168) make/take text 
messages on average between 1- 10 times per day. A little under one-third (28%; n=90) state they make or take text 
messages on average more than 21 times per day. Local communication between family, friends and colleagues is 
mainly done through texting (77%; n=244), cell phone calls (75%; n=237), and e-mail (59%; n=186). 

Approximately one-third of the participants (33%; n=104) are active on Skype and 73% (n=231) of the 
students are active on social networking sites on average 1-2 times per week. More than one -half of the participants 
(53%; n=166) report their quality of life (especially their social life) has greatly increased due to texting and/or 
visiting and/or posting on a social networking site. 

The majority of the participants believe it is socially acceptable/appropriate to make or take cell phone calls 
in a restaurant (73%; n=232) and to text when either alone (92%; n=291) or with friends and/or colleagues (71%; 
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n=223). The majority of students also agree it is appropriate to make or take cell phone calls in a doctor's 
(physician, dentist, etc.) waiting room whether you are the only person present (81%; n=256) or if other people are 
waiting (58%; n=184). Eighty-five percent (n=270) of the students report it is appropriate to text in a doctor's 
waiting room. More than one-half (56%; n=177) of the participants believe it is also appropriate to make or take cell 
phone calls in public places such as libraries, museums, and/or using public transportation (busses, trains, etc.). 

Concerning cell phone use in the university, 90% (n=284) of the students believe it is socially 
inacceptable/inappropriate to speak “even in a low tone” during a university lecture and in class discussions or group 
work (86%; n=272). Approximately two-thirds of the students believe cell phones should not be turned off during a 
university lecture (62%; n=197) or class discussion or group work (64%; n=203), but almost 90% (n=277) report 
that cell phones should be on vibrate mode during both lecture and class work. 

Albeit a majority of the students report it is socially unacceptable/inappropriate to text during both 
university lecture and class discussion/group work, approximately one-third believe it is appropriate to text during 
lecture (33%; n=103) and class work (37%; n=l 18). However, two-thirds of the students report they "cannot" 
easily text and listen to lectures (67%; n=211) or text and make and take cell phone calls while working 
cooperatively with colleagues (78%; n=248) without missing any information. Two-fifths of the students report that 
texting during both class lectures, discussions, and/or group work (44%; n=138) and meetings (40%; n=127) help 
relieve boredom. 

More than one-half (52%; n=164) of the students wish the educational system would create instructional 
strategies that utilize cell and/or smart phones. Sixty percent (n=189) of the students believe their teachers and/or 
university professors are as techno-sawy as themselves (three-fourths of the students rate their skill level in 
technology as "average" which is defined as proficiency in e-mail, cell phone, texting, and several applications 
including accessing a social networking site) and a little more than one-fourth (28%; n=87) of the students report 
that texting has increased their ability to think and or problem solve. 

Almost one-half (47%; n=148) of the participants believe they are "multi-taskers" and can easily text and/or 
make cell phone calls in any environment. Twelve percent (n=27) of the students report they frequently text while 
driving a car. 

Although 87% (n=274) of the students report they are very courteous and always speak in a low tone when 
they make or take cell phone calls, 68% (n=215) believe most people are discourteous and talk in loud tones when 
using their cell phones. Almost three-fourths (72%; n=228) of the participants believe there should be 
rules/established etiquette for where and when people can use cell phones. 

Linkages 

Significant positive and negative correlations were found in the areas of ethnicity, gender, and age. 

Ethnicity 

A positive correlation was found between ethnicity and skill level in technology (r = .127, p<.05). This suggests 
that ethnicity (cultural background) plays a positive role in the students’ perception of their skill level in technology as a 
majority of the students rated their skill level as average. Several positive correlations were also found between ethnicity 
and cell phone use during university lectures (r = .186, p<.01), university class discussions (r = .173, p<.01), texting during 
university lectures (r = .128, p<.05) and class discussions (r = . 120, p<.05). This suggests that all ethnic groups represented 
in the study strongly believe it is socially unacceptable/inappropriate to make cell phone calls or text during university 
lectures, university class discussions, and group work. Positive correlations were also found in texting in restaurants when 
alone (r = .188, p<.01), using cell phones in the doctor's office when other people are present (r = .172, p<.01), and in 
public places (r = .201, p<.01). All ethnic groups believe it is socially acceptable/appropriate to text in restaurants when 
alone, and make and take cell phone calls in public places such as doctor's waiting rooms, libraries, museums and public 
transportation. 

Gender 

Gender is positively correlated with becoming a member of a social networking site (r = .183, p<-01), 
participating on a social networking site (r = .138, p<.05), and viewing people as discourteous and speaking in loud tones 
when making or taking a cell phone call (r = .156, p<-01) . Both males and females are members and weekly participants 
of social networking sites, and view people as discourteous when using cell phones. Gender is negatively correlated with 
texting during university class discussions (r = -.119, p<.05), suggesting that males believe it is socially acceptable to text 
during class. 

Age 

There are numerable significant positive and negative correlations with age. Several positive correlations were 
found between age and cell phone use during university lectures (r = .146, p<.01), university class discussions (r = .178, 
p<.01), and communicating through e-mail (r =.114, p<.05). All age groups strongly believe it is socially 
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unacceptable/inappropriate to make cell phone calls or text during university lectures and university class discussions and 
group work; and e-mail is utilized to communicate with local and international family, friends, and colleagues. There are 
also positive correlations with perceiving oneself as courteous in cell phone use (r = .135, p<.05) and the need to establish 
rules of etiquette (r = .112, p<.05). All age groups perceive themselves as being very courteous and always speaking in a 
low tone when making or taking a cell phone call and believe there should be rules/established etiquette for where and 
when people can use cell phones. 

Age is negatively correlated with texting per day (r = .-248, p<.01), texting during university class lectures (r = - 
.203, p<.01), and university class discussions (r = -.149, p<.01). Younger students tend to text more per day, and text more 
during both university lectures and class discussions. Age is also negatively correlated with missing information from both 
university lectures (r = -.221, p<.01) and cooperative work (r = .111, p<.05). Younger students will text and miss more 
information from university lectures and cooperative work. Age and texting to relieve boredom in both class (r = .172, 
p<.01) and meetings (r = .165, p<.01) are also negatively correlated. Younger students are more likely to become bored 
during class lectures, discussions, group work and meetings and will therefore text. 

Age is also negatively correlated with both texting (r = -.120, p<.05) and cell phone use (r = -.137, p<.01) in a 
doctor’s waiting room when other people are present. Younger participants will tend to text more and make or take cell 
phone calls in a doctor’s waiting room. 

Age and texting while driving (r = -.128, p<.05) are negatively correlated. Younger students have a tendency to 
text while driving. Age and quality of life due to texting (r = -.125, p<.05) are also negatively correlated. Younger students 
believe then quality of life/social life has greatly been enhanced due to texting and visiting or posting to social networking 
sites. 

t-Tests 

There were no significant mean differences between gender or student status. 

Discussion and Implications 

In agreement with prior research, the data produced three key themes on generational differences (Rosenfeld & 
O’Connor-Petmso, 2010), curricular change, and the need to establish rules/etiquette on cell phone use (O’Connor-Petmso 
& Wei, 2011). Generational differences are seen in both cell phone use and texting. Younger students (approximately 13- 
39 years old - with the majority of the sample located in the 20-29 year old age range, followed by the 30-39 year old age 
range) believe it is socially appropriate to make or take cell phone calls in doctors’ offices when other people are present, 
send more text messages per day, text during both university lectures and group work, and text while driving. Younger 
students also believe their quality of life - especially their social life - is greatly increased due to texting and participating 
on social networking sites. 

The need for curricular change, specifically the need for teachers and professors to include cell and smart phones 
in their instructional activities and the need to formally establish rules/etiquette on cell phone use in an array of 
environments emerged in all ethnic and age groups. A plethora of research (Marsh & Shavelson, 1985; Walberg, Fraser, & 
Welch, 1986) evinces motivational teaching strategies, self-concept, and self-esteem as predictors of achievement. The 
results of this study show that almost one-third of the students perceive texting as increasing their ability to think and/or 
solve problems and situations and more than 50% of the students wish their teachers and professors would use e- 
communication tools in their coursework. 

Similarly, without formally accepted rules on cell phone use in public places, the quandary of discourteous 
people who talk in loud tones on cell phones will not abate and could increase with new and emerging technologies. If one 
perceives his/her own behavior with cell phones as courteous and speaking in low tones, as the participants of this study 
overwhelmingly did, the problem could continue. In addition, accepted rules on cell phone use will help students focus on 
university lectures and class work and meetings and hopefully prohibit younger students from texting while driving. 
Although many states mandate the use of “hands-free cell phones while driving,” present research indicates that young 
people continue to text while they drive. Perhaps more drastic measures are needed to educate young people of the dangers 
associated with texting and driving. 

Based on the data and prior research, the authors propose some guidelines on Cell Phone Etiquette. The research 
indicates generational differences between younger students and older students. Younger students seem to favor the use of 
cell phones in public places and in school. Whereas older students are more likely to turn off or set cell phones to vibrate 
in public places, those who choose to use a cell phone in public places should be especially courteous and sensitive to 
others. They should speak in a low voice and avoid topics that are personal in nature so as not to offend others. If possible, 
they should remove themselves from public areas (including mass transit) where individuals gather. If they cannot get 
away from others, perhaps they should not make or take the call. 
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Texting should not be permitted during university lectures, during class work and/or meetings if it is annoying to 
others. The results of this study indicate that younger students miss information when they are texting. If they paid closer 
attention to the class discussion rather than paying attention to their cell phones, perhaps they would not miss information. 
Participation in a class discussion can also help to ward off boredom, another reason given for texting. 

Admittedly, there are times that cell phone technology is useful in a classroom. Smart phones are excellent tools 
for looking up information on the Internet and they are fine for learning why a classmate has not arrived in class. There are 
many ways cell phones can be used to enhance learning. However, one of the authors tells her students that if she sees a 
student with a cell phone when none is called for, (especially if someone texting or using a cell phone for something 
unrelated to class) this can have a negative effect on his/her grade. This pronouncement seems to have cut down the 
number of inappropriate uses of the cell phone in class. 

In conclusion, based upon the aforementioned research data and prior research, professors need to find ways to 
implement cell phones into course curriculum and simple concrete rules need to be set in force in order to adhere to 
pragmatic cell phone etiquette in diversified environments. 
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Abstract 

This research examines nonnative-English-speaking students’ perceived effectiveness of asynchronous embedded 
audio feedback in an online course when feedback was provided by a nonnative-English-speaking instructor versus a 
native-English-speaking instructor. A quasi-experimental study was conducted to investigate students’ perceptions 
of using the technique when participating in an asynchronous online discussion. The study has implications for 
instructors and designers responsible for creating online learning environments that involve nonnative-English- 
speaking students. 



Introduction 

The role of instructional feedback in asynchronous online environments and specifically in asynchronous 
online discussions is the most important because students participating in asynchronous online discussions feel 
stressed, disconnected, or left behind when they do not receive any feedback on their posting (Dabbagh, 2002). 
Feedback in asynchronous online discussions supports students’ success to complete the online course, i.e., the lack 
of feedback is viewed as one of the reasons why students drop the courses (Ertmer, Richardson, Belland, Camin, 
Connolly, Coulthard, Lei, & Mong, 2007). Flowever, there are several problems with delivering feedback in 
asynchronous online discussions. The most common problem is that students are unable to understand feedback 
comments and to interpret them correctly (Fliggins, 2000; Quinton & Smallbone, 2010). Understanding instructional 
feedback has become crucial in times when asynchronous online environments allow students to enroll who are 
geographically dispersed and who come from different cultures and countries (Shih & Cifuentes, 2003; Zhang & 
Kenny, 2010). It is becoming a common practice that university online courses enroll international and transnational 
or nonnative-English-speaking students; such online courses are offered in countries other than those where students 
are located (Zhang & Kenny, 2010). 

Accordingly, recent research on university online courses has extensively examined the impact of 
asynchronous online discussion on nonnative-English-speaking students; the majority of studies have concentrated 
on American higher education online courses in which students were non-native speakers of English (Biesenbach- 
Lucas, 2003). However, the majority of research findings have shown that nonnative-English-speaking students 
enrolled in American higher education online courses and participating in asynchronous online discussions face 
serious challenges related to language proficiency leading to misinterpretation of instructional comments (Olesova, 
Yang, & Richardson, 2011; Shih & Cifuentes, 2003). Likewise, Zhang and Kenny (2010) found evidence that the 
language barrier might have prevented nonnative-English-speaking students from contributing to online discussions 
as often as they would have desired (Zhang & Kenny, 2010). Moreover, their English language proficiency could 
limit nonnative-English-speaking students’ ability to be productive in the online courses, e.g., the students need 
more time than their English-speaking peers to read and compose messages (Zhang & Kenny, 2010). In addition, as 
long as asynchronous online discussions are built on written communication on account of lack of visual and aural 
clues, language skills seem to be one of the most significant skills required to participate in asynchronous online 
discussions (Black, 2005). 

To overcome the limitations of text-based communication, research has shown the importance of the 
instructor’s role in facilitating online discussions for successful online learning (Anderson, Rourke, Garrison, & 
Archer, 2001; Biesenbach-Lucas, 2003; Swan, 2003). Indeed, the instructor’s role to provide guided instruction, 
encourage critical reflection, and give constructive feedback may enable students to overcome difficulties of text- 
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based online communication (Biesenbach-Lucas, 2003). Yet, to increase both the verbal and nonverbal cues of 
asynchronous interactions, studies have proposed using asynchronous audio, specifically, instructional audio 
feedback (Ice, Curtis, Phillips, & Wells, 2007). Audio feedback, defined as a recorded message in online instruction, 
has been viewed as a means to overcome the lack of clarity in text-based communication. Moreover, audio 
feedback, when embedded in a student’s written documents, has demonstrated that it can strengthen the instructor’s 
ability to affect learning and to generate more personalized communication with students (Ice et ah, 2007). 

Interest in the effectiveness of using audio feedback in teaching began in the 1960’s (Coleman, 1972; 
Logan, Logan, Fuller, & Denehy, 1976; Lumsden, 1962; McGrew, 1969; Moore, 1977; Tanner, 1964). The first 
empirical studies on using audio feedback were conducted with native speakers of English in the field of English 
Composition in high school and revealed that using audio feedback was an effective technique to improve first 
language writing (Coleman, 1972). More recent studies for native speakers (e.g., Berner, Boswell, & Kahan, 1996; 
Mellen & Sommers, 2003; Sipple, 2007) also found that audio feedback positively affected students’ perception of 
their motivation and self-confidence in a traditional writing class. 

Today, due to the development of distance education and an increased number of online courses, 
researchers’ and practitioners’ interest in using audio feedback in asynchronous online environment has increased. 

In the field of distance education, research results have shown that students receiving instructional audio feedback 
described their experience as personal, enjoyable, complete, and clear (Kirschner, van den Brink, & Meester, 1991). 
Studies exploring audio feedback in online courses indicate that this technique increases both retention of content 
and student satisfaction as compared to asynchronous text-only feedback (Ice, Swan, Kupczynski, & Richardson, 
2008). In addition, a research project funded by the Higher Education Authority (HEA) in the United Kingdom also 
supported the effectiveness of using this technique in online courses and students’ positive perceptions in receiving 
audio-recorded feedback (McFarlane, 2009; Nortcliffe & Middleton, 2007). 

Asynchronous embedded audio feedback was perceived to be more effective for conveying nuance, it 
increased engagement and retention of context, it enhanced learning community interactions, and it was associated 
with the perception that the instructor cared more about the student (Ice et al., 2007; Oomen-Early, Bold, Wiginton, 
Gallien, & Anderson, 2008). Conveying nuance is very important in asynchronous online discussions, as Swan 
(2003) explains, because real-time negotiation of meaning is impossible among instructors and students separated by 
space and time, making clarity of meaning even more imperative in online classes. 

Research on the effectiveness of audio feedback for nonnative-English-speaking students in asynchronous 
online courses has found evidence that audio feedback helped nonnative-English-speaking students improve their 
second language skills (Hsu, Wang, & Comae, 2008). The nonnative-English-speaking students listened to natural 
oral speech; audio feedback helped them relate to the mistakes that they had in their assignment (Hsu et ah, 2008). 
Overall, the majority of studies have examined the effectiveness of audio feedback in asynchronous online 
environments when it was provided by a NS instructor (Hsu et ah, 2008; Ice et ah, 2007; Oomen-Early et ah, 2008). 
However, no studies have been done to examine whether audio feedback can become an effective technique when it 
is provided by a NNS instructor in asynchronous online environments (Ice, 2008). It is still not clear whether audio 
feedback can increase nonnative-English-speaking students’ online learning and satisfaction as they might be at a 
disadvantage due to their “linguistic difference” and “cultural otherness” (Gunawardena &McIsaac, 2004). 

Therefore, this study examined how embedded audio feedback was perceived by nonnative-English- 
speaking students enrolled in an online course in spring 2011. Specifically, the study investigated: 1) the possible 
differences in students’ scores on perceived effectiveness of the technique when it is provided by a nonnative- 
English-speaking instructor (NNS) versus a native-English-speaking instructor (NS), 2) the possible differences in 
students’ scores on perceived effectiveness of the technique by the students’ level of English language proficiency, 
and 3) the possible interaction effect on perceptions by instructors’ language background and participants’ level of 
language proficiency. 



Methods 

Participants and Setting 

The participants (n=69) were nonnative-English-speaking students from a university in Russia. The 
participants were enrolled in an online course delivered in English. The two instructors from a large Midwestern 
university included a graduate student as the NNS (also Russian), and a graduate student who is the NS. The 
language and ethnicity of the NS was determined through criteria of the ethnicity, birthplace and language spoken. 

Both instructors received training on how to provide both types of feedback and how to use Adobe Pro to 
embed the audio feedback. Thirty-six participants were randomly assigned to the group of NS instructor including 
16 participants at a high language level (TOEFL score higher than 513) and 20 were at a low language level 
(TOEFL score lower than 513). Thirty-three participants were assigned to the group of NNS instructor including 15 
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participants at a high language level and 18 at a low language level. However, only 29 participants in the NS 
instructor’s group completed the course, with 15 participants at high level of language proficiency and 14 
participants at low level of language proficiency. Twenty five participants in the NNS instructor’s group with 15 
participants at high level of language proficiency and ten participants at low level of language proficiency were able 
to finish the course. Each group received two text-based and two audio feedback responses. The online survey was 
administered at the end of the study. A total of 55 participants (85.94 % response rate) completed the online survey. 

Data Collection and Analysis 

The seven-item 5-point Likert scale audio feedback survey (Ice, 2008) was used to examine perceptions on 
audio feedback. The survey asked participants to rate their agreement/disagreement regarding how they perceived 
audio feedback compared with how they perceived text-based comments based on the following items: clarity of 
instructor’s intent, their involvement in the course, their motivation, their retention of the course, and instructor’s 
care. The audio feedback survey had high internal consistency reliability with a Cronbach’s alpha coefficient 
reported of .87 with inter-item correlations ranging from .70 down to .32. Participants’ responses on the audio 
feedback survey (Ice, 2008) were analyzed using descriptive statistics, the Mann- Whitney Test and a two-way 
between groups ANOVA. 



Results 

The participants reported that they were satisfied with asynchronous embedded audio feedback (M= 3.63). 
The items on the clarity of instructor’s voice (M=3.98) and personality of audio feedback (Af=3.85) had the highest 
means across all the items. However, the responses on the items on motivation (M= 3.27) and retention (Af=3.16) 
perceptions had the lowest means across all the items. 

The participants’ perceptions for the type of feedback were compared, based on the frequencies of 
numbers to each individual survey item, with regard to working in a NS instructor’s group and working in an NNS 
instructor’s group at the high and low levels of language proficiency. 



Item One “When using audio feedback, inflection in the instructor’s voice made his / her intent clear” 

The descriptive analysis of the results and findings indicated that the majority in the NS (66%) and NNS 
groups (72%) would strongly agree and agree that, when using audio feedback, inflection in the instructor’s voice 
made his/her intent clear. Similarly, the majority at the high (74.1%) and low (64.3%) levels of language proficiency 
also self-reported that they would strongly agree and agree that the instructor’s voice made intent clear when using 
audio feedback. 




Figure 1. Percent of responses to item one by instructors and language proficiency. 



Item Two “The instructor’s intent was clearer when using audio than text” 

The survey data revealed a high percentage of the participants believing the instructor’s intent was clearer 
when using audio rather than text. The statistical analysis using frequencies showed the majority of the participants 
gave preferences towards audio feedback by the instructors’ language background and level of language proficiency. 
Of the 29 participants working in the NS instructor’s group, 53% agreed and strongly agreed that the instructor’s 
intent was clearer when using audio rather than text. The findings from the NNS instructor’s group showed that 64% 
agreed and strongly agreed on the same survey item. Similarly, the analysis by the participants’ level of language 
proficiency also revealed that 52% at the high level and 64% at the low level gave preferences to audio feedback on 
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the item of the clarity of instructional intent. However, the analysis indicates that 40% in the NS instructor’s group 
and 41% at the high level of language proficiency were neutral on the same item. 





Figure 2. Percent of responses to item two by instructors and language proficiency. 



Item Three “Audio comments made me feel more involved in the course than text based comments” 

The data by the instructors’ language background and level of language proficiency on the item whether 
audio comments made the participants feel more involved in the course than text based comments revealed the 
participants’ positive perceptions. Out of 29 participants in the NS instructor’s group, 50% self-reported that they 
agreed and strongly agreed that audio feedback made them feel more involved in the course. Similarly, 60% in the 
NNS instructor’s group also agreed and strongly agreed on their feeling for involvement in the course. Further, the 
analysis by the level of language proficiency revealed that 56% at the high level and 54% at the low level preferred 
audio feedback for their feeling to be involved in the course. However, more than 30% in both groups at both levels 
of language proficiency indicated a neutral position concerning the item on the course involvement when receiving 
audio feedback. 




Figure 3. Percent of responses to item three by instructors and language proficiency. 



Item Four “Audio comments motivated me more than text based comments” 

The statistical analysis using frequencies of numbers showed that the participants had different preferences 
on the item whether audio comments motivated them more than text based comments. The majority (54%) in the NS 
instructor’s group responded more positively towards audio feedback in relation to motivation while only 32% in the 
NNS instructor’s group agreed on this item. Further, the analysis for the NNS instructor’s group showed that 36% 
were neutral and 32% disagreed/strongly disagree that audio feedback motivated them more than text based 
comments. As for the level of language proficiency, the percentage of the participants at the low level rated the 
motivation item higher than the high level (47% versus 41%). However, 32% of the participants at the low level and 
26% at the high level disagreed and strongly disagreed that audio comments motivated them more than text based 
comments. Finally, 33% at the high level and 21% at the low level were neutral on the item that audio comments 
motivated them more than text-based comments. A Mann-Whitney test revealed no significant difference in the 
feedback preferences of the NS instructor’s group (Md = 4.00, n = 30) and the NNS instructor’s group (Md = 3.00, n 
= 25), U— 307.000,/? = .24. As for the level of language proficiency, a Mann-Whitney test also did not reveal any 
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significant difference in the feedback preferences of the high level ( Md = 3.00, n = 27) and the low level ( Md = 
3.00, n = 28), U= 3358.500 , p = .74. 




disagree agree 



INSTRUCTORS 
■NS (n=30) 
^NNS (n=25) 




disagree agree 



PROFICIENCY 
■HIGH (n=27) 
^LOW ( n =28) 



Q4 



Q4 



Figure 4. Percent of responses to item four by instructors and language proficiency. 



Item Five “I retained audio comments better than text based comments” 

Similar to the results on the motivation item, the responses on the question whether the participants retained 
audio comments better than text based comments, 50% of the participants in the NS instructor’s group responded 
that they agreed and strongly agreed while only 32 % of the participants in the NNS instructor’s group agreed on the 
retention item. However, similar to the results on the motivation item, almost 30% of the participants in both groups 
gave negative responses regarding retention from audio comments. In addition, 40% of the participants in the NNS 
instructor’s group were neutral that they retained audio comments better than text based comments. The analysis for 
the level of language proficiency shows that the percentage of those who agreed and strongly agreed on the retention 
item from audio comments was higher at the low level of language proficiency (46% versus 37%). However, the 
percentage of neutral responses on the retention item was higher among the participants at the high level of language 
proficiency (37% versus 21%). Finally, the percentage of negative responses was higher at the low level of language 
proficiency (32% versus 26%). A Mann- Whitney test revealed no significant difference in the feedback preferences 
of the NS instructor’s group (Md = 3.50, n = 30) and the NNS instructor’s group (Md = 3.00, n = 25), U= 333.500, 
p = .47. As for the level of language proficiency, a Mann- Whitney test also did not reveal any significant difference 
in the feedback preferences of the high level (Md = 3.00, n = 27) and the low level (Md = 3.00, n = 28), U= 

366.500, ^ = .84. 





PROFICIENCY 
■HIGH (n=27) 
ULOW (n=28) 



Figure 5. Percent of responses to item five by instructors and language proficiency. 



Item Six “Audio comments are more personal than text based comments” 

The statistical analysis demonstrated that the majority of participants indicated that they strongly agreed or 
agreed that the audio comments were more personal than the text based comments. This was true based both on 
instructors’ language background (60% in NS and 76% in NNS) and on the language proficiency level (63% at the 
high level and 71% at the low level). 
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Figure 6. Percent of responses to item six by instructors and language proficiency. 



Item Seven “Receiving audio comments made me feel as if the instructor cared more about me and my work 
than when I received text based comments” 



The statistical analysis showed that the majority of participants in the NS (60%) and NNS (56%) groups 
self-reported that they strongly agreed and agreed that receiving audio comments made them feel as if the instructor 
cared more about them and their work than when they received text based comments. Similarly, the majority of the 
participants (68%) at the low level of language proficiency gave preferences to audio comments while only 49% of 
the participants at the high level agreed on the item and 37% stayed neutral on their preferences. 




Figure 7. Percent of responses to Item Seven by instructors and language proficiency. 



In order to answer the research question of whether or not there was a significant difference in scores on the 
participants’ perception by instructors’ language background and participants’ level of language proficiency, two- 
way between-groups ANOVA on the survey scores was conducted. The main effect for instructors’ language 
background was not statistically significant, F ( 1, 51) = .20, p = .66. The main effect for participants’ level of 
language proficiency was also not statistically significant, F (1, 5 1) = .28, p = .60. 

To answer whether or not there was any interaction between scores on feedback perception by the 
instructors’ language background and the participants’ level of language proficiency, a two-way between-group 
ANOVA was run. Results (Table 1) indicated there was no statistically significant result for interaction effect 
between the instructors’ language background and participants’ level of language proficiency, Ail, 51) = 3.83, p = 
.06 (Figure 8). 
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Table 1 



Two- Way Between-Groups ANOVA: Effect of Instructors ’ Language Background and Participants ’ Levels of 
Language Proficiency > 



Source 


Type III Sum 
of Squares 


df 


Mean 

Square 


F 


P 


Instructors 


0.09 


1 


0.09 


0.20 


0.66 


Proficiency 


0.12 


1 


0.12 


0.28 


0.60 


Instructors * Proficiency 


1.71 


1 


1.71 


3.83 


0.06 



PROFICIENCY 

HIGH 

- • • LOW 



Figure 8. Plot of interaction between levels of language proficiency and instructors’ language background on 

participants’ perception. 

Further, post hoc power analysis was run to calculate power for the interaction effect. The priori power 
analysis was conducted using the G*Power software given the two-way between-groups ANOVA, the alpha level of 
significance at .05, expected effect size .40, and the total sample size of 55. The results revealed that the power for 
the interaction effect was .13, indicating only a 1.3% probability of finding a statistically significant result between 
the instructor’s language background and the participants’ level of language proficiency. The power was relatively 
low and the replication of the study with larger sample size will help to evaluate the current findings. 

Discussion 

The findings from the survey overall corroborated previous studies (Ice et al., 2007; Ice, 2008; Ice et ah, 
2008; Oomen-Early et ah, 2008) showing that nonnative-English-speaking students in this study also preferred 
receiving asynchronous embedded audio feedback over text-based comments (M= 3.63, SZ)=0.30) but their 
perception may vary by level of language proficiency (high and low) and instructors’ language background (NS and 
NNS). The findings from the survey by instructors’ language background and participants’ level of language 
proficiency also revealed that overall the participants in the NS instructor’s group (Af=3.07, .S'O — 0.70) and the NNS 
instructor’s group (Af=3.29, SD= 0.65) at the high level of language proficiency (M= 3.14, £0=0.65) and the low 
level of language proficiency (A/=3.21, £0=0.7 1 ) preferred receiving audio feedback over text-based comments. 

Furthermore, the descriptive analysis of participants’ responses presented the evidence that the participants 
in the NNS instructor’s group rated their perception for audio feedback over text-based comments higher than the 
participants in the NS instructor’s group. Interestingly, the survey responses provided an unexpected finding on the 
participants at the low level of language proficiency. In addition, responses on the questions whether audio feedback 
motivated them and if they retained audio comments better than text-based feedback achieved the lowest scores 
consistently across the groups of NNS and NS instructors at both levels of language proficiency (high and low). 
Although no statistical differences were revealed between the NNS and NS instructors’ groups and levels of 
language proficiency on perceptions for the type of feedback delivery method among the nonnative-English- 
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INSTRUCTORS 
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speaking students it is believed, as is argued in the publication Effective Practice in a Digital Age (2009), that audio 
feedback was perceived positively by the nonnative-English-speaking students in both groups (NNS and NS) and at 
both levels of language proficiency. 

Clarity 



The majority in both groups (NNS and NS) at the high and low levels of language proficiency positively 
rated both survey items on clarity. This finding supports previous studies on audio feedback, suggesting that the 
nonnative-English-speaking students’ prefer audio comments because they are clearer and more understandable than 
written comments (Ice, 2008; Nortcliffe & Middleton, 2008). These results support recent literature suggesting that 
students preferred asynchronous audio feedback as compared to traditional text-based comments because of 
understanding nuances and clarity of meaning which is very important for communication in an asynchronous online 
environment (Ice et al., 2007). These findings are also consistent with recent literature on audio feedback indicating 
that students preferred audio feedback for better understanding of how to improve their work and for providing 
greater detail than written comments (Ice, 2008). Therefore, audio feedback could be an effective technique when it 
is provided by NS and NNS instructors and for both levels of language proficiency in asynchronous online 
environments. In this sense, audio feedback can overcome the lack of clarity in text-based communication among 
the nonnative-English-speaking students, especially when they communicate with their instructor who is a native 
speaker of English (Ice et al., 2007; Quinton & Smallbone, 2010). 

However, findings from both clarity items also showed that the students in the NS instructor’s group rated 
both items lower than the NNS instructor’s group. Therefore, the low rating of the NS instructor may corroborate 
with findings from previous studies (Zhang & Kenny, 2010) indicating that the nonnative-English-speaking students 
might have experienced problems understanding their instructor who was a native speaker of English. Interestingly, 
the participants at the low level of language proficiency rated the item on the clarity of the instructor’s intent higher 
than the participants at the high level of language proficiency. One possible reason for this is that audio feedback 
can assist the nonnative-English-speaking students in overcoming drawbacks of text-based feedback to provide 
clarity of meaning for the nonnative-English-speaking students, especially at the low level of language proficiency. 
These findings can be compared to Price and Holman’s (1996) investigation of minority Hispanic students in the 
U.S. The researchers found that Spanish-speaking bilingual students responded more enthusiastically to the taped 
feedback than Anglo students did. 

Involvement, Personalization and Instructor’s Care 

The results of this study showed that the majority of the nonnative-English-speaking students preferred 
audio feedback because it is personal and it makes them feel more involved in the online course. The findings of this 
study support the literature that indicates audio feedback can increase students’ feelings of involvement in the online 
courses and the more personalized communication with their instructors. They prefer audio feedback because it is 
more personal than text-based comments and audio feedback can increase the feeling of the instructor’s concern for 
the students (Hsu et al., 2008; Ice et al., 2007). Interestingly, about 70% of the participants at the low level of 
language proficiency rated the item on instructor’s care much higher than the participants at the high level of 
language proficiency. So, it seems that audio feedback could be effective for the nonnative-English-speaking 
students at the low level of language proficiency in creating the feeling of the instructor’s care when receiving audio 
comments. 

Motivation and Retention 

Overall, the nonnative-English-speaking students rated the motivation and retention items positively; the 
findings may support previous studies revealing that audio feedback helped to enhance students’ motivation and 
retention (Ice, 2008). However, these items scored consistently lower than the other survey items across both 
instructors and both language proficiency levels. Surprisingly, almost half of the participants at the low level of 
language proficiency had more positive perception that they retained audio comments better than text based 
comments than the participants at the high level of language proficiency. Then, for the nonnative-English-speaking 
students in the NS instructor’s group perceived more motivation and retention than those in the NNS instructor’s 
group. These findings are consistent with previous studies revealing that for nonnative-English-speaking students 
perceived that audio feedback provided by a native speaker of English reinforced their assignment with an authentic 
listening exercise which created an extra motivating factor (Hsu et al., 2008). It could be assumed that the 
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participants in the NS instructor’s group used audio feedback from their instructor who was native speaker of 
English as an additional instructional tool to develop their listening skills. 

To summarize, it could be assumed that even the groups (NNS and NS) and the levels of language 
proficiency (high and low) were not significantly different in their perceptions of the types of feedback; the findings 
suggest that the level of language proficiency and instructors’ language background can have an impact on the 
perceived effectiveness and perception of the technique during the online course. 

Implication and Future Research 

The nonnative-English-speaking students had positive feelings of engagement in the course and the 
instructor’s care when receiving audio feedback compared to text-based feedback. This can imply that audio 
feedback allows instructors to help nonnative-English-speaking students in a constructivist learning environments 
(Mory, 2004). In addition, audio feedback can provide clearer and more personal feedback than text-based feedback. 
Thus, the finding on perceived feeling of involvement implies that using audio feedback for the nonnative-English- 
speaking students can reinforce the sense of “being there” in order to remove transactional distance when teaching 
and learning occur in separate locations (Moore, 2007). 

When receiving audio feedback, the nonnative-English-speaking students can benefit from the NNS 
instructors with the same native language, ethnicity, and cultural background to increase the quality of online 
posting because of the instructor’s familiar accent and the structure of English used (e.g., the structure of English in 
this study provided by the NNS instructor was based on English to be taught in Russia). Then, when providing audio 
feedback for nonnative-English-speaking students, the NS instructors can help increase the students’ motivation, 
retention, and perceived feelings of the instructor’s care. 

The findings that the nonnative-English-speaking students at the low level of language proficiency rated the 
motivation and involvement items lower but retention item much higher than the students at the high level of 
language proficiency need further investigation to examine how a student’s level of language proficiency can 
enhance perceived motivation, retention and involvement when receiving audio feedback in an asynchronous online 
environment. Further research is needed to investigate other types of feedback (e.g., combined written and audio 
feedback) to determine the value of text-based feedback and audio feedback for EFL students (Ice et ah, 2010). 

Other types of formats for providing audio feedback are also needed, for example, Camtasia and Captivate. 



Limitations 

Some limitations affected the outcomes of this study. First, the sample was not random. Then, the 
participants were chosen from only one program and the lack of participants’ variability can make the application of 
this research finding in other disciplines and other institutions difficult. Finally, the small sample size limited the 
examination of the interaction effect. 

Despite several limitations, this study’s findings have important implication for moving the investigation of 
audio feedback effectiveness for nonnative-English-speaking students forward and identifying best practices in 
asynchronous online environments. This study suggests the importance of understanding how nonnative-English- 
speaking students ‘level of language proficiency and the instructors’ language background can impact the students’ 
perceptions. This is especially true when communication conducted in English, specifically when students receive 
audio comments recorded in English. Nevertheless, this study suggests that embedded audio feedback provided for 
nonnative-English-speaking students can be viewed as an effective technique to increase the perceived effectiveness 
of the technique in an asynchronous online environment. 
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Designing Learning Object (LO) can be very challenging when the students are those who come from the special 
needs background. Research has been conducted and has identified the need for designing a specific LO for dyslexic 
children in learning multiplication topic. The objective of this research is to describe the development and evaluation 
process of the proposed Interactive Multimedia Learning Object (IMLO) in three phases of (1) Analysis, (2) 
Development, and (3) Evaluation. Analysis phase identifies the user needs through user analysis and content 
analysis. The development phase describes the proposed teaching strategy implemented in IMLO. The evaluation 
phase elaborates the dyslexic children evaluation. RLO Student Evaluation form was used to ask and observe the 
students on the use of IMLO. The result reveals positive feedback in terms of acceptance, attention, usability, 
narrative content, interactivity, activity and comprehension. 

Keywords: Dyslexia, Learning Object, Multimedia components, Usability. 

1. Introduction 

1.1. Background of Study 

Dyslexia is defined as a specific type of learning disability involving a severe impairment in reading ability 
which affects and disrupts a person’s language development and functioning (Carol & George, 1996). The 
individual with dyslexia can actually become confused when several instructions are given at the same time, and 
will usually have a poor-short-term memory, difficulty with directional orientation, such as telling right from left 
and map reading (Reid, 2005). 
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There were about 314,000 school-going children in Malaysia who have dyslexia in the year 2005 and the 
number may have actually increased. Learning difficulties experienced by these children will contribute to future 
educational, social, psychological, health and employment disadvantages. For these children, the long term 
consequences of having experienced failure at school are very alarming. Knowing the importance of basic skill of 
literacy (known as the 3M, which refers to Membaca (Reading), Menulis (Writing) dan Mengira (Counting)), 
dyslexic children may experience serious problem on their survival skills, such as counting money, recognizing bus 
number and house address. One of the contributing factors that children do not have the 3M is cognitive problem 
such as dyslexia. Komala (2004) reported that there were five percent (5%) dyslexia case identified within any 
community or at every twenty (20) students in Malaysia. On the other hand, the President of Malaysia Social 
Harmony Association, Nordin Ahmad (2005) added that his organization has conducted a research and found that 
about ten to fifteen percent ( 1 0%-l 5%) primary school students, have dyslexia. 

Mohd. Sharani Ahmad (2004) recommended some intervention approaches and techniques that can be 
employed to help SLD students, including educational intervention (strategies, activities & environment) technical 
intervention (learning packages & voice printing programs) and medical intervention (Drug therapy and diets). 
According to Nigel (n.d.) explored ways in which multimedia can be used to enhance the accessibility of the learning 
environment. Multimedia has the potential to reduce or even remove such problems. For example, learning materials, 
containing text, can be supplemented with and/or represented in graphical and auditory forms. He also added that 
dyslexic students are able to comprehend meaning from what is being spoken about a picture. Having learning 
materials delivered in this way can reduce the difficulties dyslexic students have in recognizing or confusing between 
letters or familiar words. Furthermore, hearing new spoken words can help dyslexic students with mispronounciations. 
It can help them to form links between what a new word sounds like and what it looks like. 

In terms of the strategy, Joan Dean (1996) suggested that children with learning difficulties need a number 
of different teaching approaches such as work which is broken down into small steps, activities which enables them 
to practice the learning they acquire to the point when it is over-learned, work which involves the stimulus of first- 
hand experience and some collaborative works. Therefore, the researcher believes that Learning Object (LO) is the 
perfect match. LO is a content that offers the opportunity to create instructional elements (content segments, process 
instructions, and affective exercises) in small parts that are reusable, scalable, and adaptable in multiple and varied 
learning contexts (Wiley, 2001). According to Hamel and Ryan-Jones (2002) A Learning Object narrowly defined, 
refers to a small, stand-alone unit of instruction that can be tagged with descriptors and stored in repositories for 
reuse in various instructional contexts. 



1.2. Problem Statement 

Early analysis was conducted to identify the use of LO at seven schools with dyslexia program in Klang 
Valley including Malaysia Dyslexia Pilot School as well as one NGO. Through site visit and telephone 
conversation, the researcher has found three research problems; (1) there is no LO available at the dyslexia schools, 
(2) No multimedia approach employed to teach the dyslexic children, (3) There is limited research done on dyslexia 
with numeracy problem in Malaysia. The same statement was agreed upon by Ishak (2008) as well as no related 
research was found in Malaysian Thesis Online (MYTO). 

1.3. Research significance 

This study describes the development and evaluation of Interactive Multimedia Learning Object (IMLO) 
for dyslexic students with numeracy problem. In specific, IMLO consists of specific topic on mathematics subject 
based on the need of the dyslexic children. It is hoped that the dyslexic children can benefit from IMLO, improve 
their learning activity, and to a certain extent,, reduce the numeracy problem. 

Realizing the importance of IMLO development, the content quality has become a concern. IMLO content 
would not just generally help the dyslexic children to learn, but it also becomes a useful tool in providing 
information, giving opportunity for the dyslexic children to interact and create a new atmosphere for knowledge 
retrieval. The interactive multimedia functions provide special instruction and feedbacks in interactive learning 
activities. The dyslexic children should be able to adapt themselves in the learning process according to individual 
preferences. According to Mortimore (2008), such situation is common amongst the dyslexics, it is called 
automaticity. 
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Thus, this research provides relevant information about dyslexia learning preferences, especially for 
learning mathematics. Teachers can benefit by using IMLO to plan towards more effective teaching strategies. In 
fact, more importantly, this research provides advice in setting standard guideline for producing such IMLO. Since 
IMLO content is carefully designed for easy-learning purpose, it can be reused for various types of learning 
disability. It is hoped that this research will contribute to the method of IMLO development and its application on 
the dyslexic children. 



2. Literature Review 

Applying multimedia in LO can help the dyslexia learning process. Keates (2000) supported the use of 
software as a learning tool. The same agreement came from Mortimore (2008) who confirmed that dyslexic students 
learn better when they are taught using midtisensory approach. The use of multimedia type of LO has been proven. 
Rosnah (2007) has developed LO Science and revealed that the LO provides educational value and value added to 
form- four Malaysian students. Another benefit of LO is proven by Kay & Knaack (2005), where they have 
developed five LOs for secondary school students and found that two thirds of the students reported that the LOs 
were beneficial, citing a motivating theme, interactivity, and found visual qualities as the most important features. 

The researcher found similar approach done by Ismail et al (2010), where a courseware (called E-Z- 
Disleksia) has been developed to accommodate the needs of dyslexic children with difficulty in reading and learning 
to read Malay language. It was made by some concerns such as (1) content structure (where six modules were 
organized based on levels), (2) navigational structure, to permit flexibility amongst dyslexic children to explore the 
content, and (3) main menu, that act as the index page, (4) sub module, the content to learn, and (5) activity, where 
the dyslexic children needs to follow the animated hand written letter. 




Figure 1 : Screenshot sample of e-z-disleksia showing the main menu and the content. The use of pure white 
background might lead to reading problem amongst dyslexic children. 

However, e-z-disleksia multimedia elements do not show solid justification in terms of the dyslexic 
children preferences. It is important to understand the dyslexic children learning style (Fadilahwati & Fattawi, 2009) 
and the multimedia components have they own role in providing instructions. To fully exploit interactive 
multimedia components to full advantage of the instruction, designer needs to be aware of how learners actually 
learn as well as the different types of learning that exist. McEwan & Cairncross (2004) believed that meaningful 
evaluation of Multimedia Learning Object requires consideration of integration of learning application into 
curriculum. They also added that attention must be given on how best to incorporate multimedia components to 
enhance learning. Previous research done by Fadilahwati et al (2011) revealed that dyslexic children needed to be 
motivated and engaged by using multimedia components such as animation, audio, graphic and text. 
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On the other hand, Ronaldi and Fadilahwati (2009) have identified the benefits of dyslexia learning content 
on mobile interactive comic. D-Mic (Dyslexia Mobile Interactive Comic) delivered learning content about 
understanding Malay verbs. D-Mic was designed as a mobile application. To use it, dyslexic children need to open 
the application under the ‘program’ menu. D-Mic content was made interactive. Dyslexic children can learn a 
specific topic ‘verbs’ on Malay language subject. 






v.l 



? 



vy** 



AMIR 

DAN 

BASIKAL 




Figure 2: D-Mic screenshot. 

D-Mic developmental process was made using strategy. Referring to Hughes et al (1996) who stated that 
multimedia (integration of text, graphics, animation, sound and full motion video) can be utilized as a focal point to 
yield attention and participation of students, D-Mic implemented animation and sound to support the text meaning. 
The researcher thought that such strategy was suitable because dyslexic children is known for having poor attention. 
Everatt et al (1999) and Goldish (1997) confirmed that dyslexics have problems in paying attention. The use of two 
words text per screen was also found to be good. Even though Pepper & Lovergrove (1999) suggested to present one 
word text at a time for easy reading, D-Mic research proved that dyslexic children was able to read the two words 
easily with the help of multimedia elements. In terms of presenting materials for the dyslexic children D-Mic 
approach was also found aligned with Reid & Green (2007), who suggested using large font size with relevant 
pictures to aid the text comprehension. D-Mic was tested on children with dyslexia in Malaysian schools. 
Observation and interviews were conducted to assess questions on usability and D-Mic was found motivating. The 
children had no difficulty reading the text, following the sequence and performing the given task. One contributing 
success factor is the D-Mic was split into small units and this ease understanding amongst the dyslexic. On another 
research, Fadilahwati et al (2010) also confirmed that using Multimedia-based Learning Object, where content is 
broken down into small steps, dyslexic children can perform the task in a short time, and lead to better learning 
process. 



3. Methodology 

This research suggests the development of Interactive Multimedia Learning Object (IMLO). According to 
Richey & Klein (2007), developmental research is the systematic study of design, development and evaluation 
processes with the aim of establishing an empirical basis of the creation of instructional and non-instmctional 
products and tools and new or enhanced models that govern their development. 
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Multiplication-of-two topic has been selected as the content to be strategized using IMLO. The process 
from topic selection until evaluation is described based on three phases (see Figure) of (1) Analysis, (2) Prototype 
Development, and (3) Evaluation. 




Figure 3. IMLO developmental process 



This research describes the assessment of IMLO by the dyslexic children. IMLO is evaluated using RLO 
Student Evaluation form as adapted from the Centre of Excellence in Teaching and Learning for Reusable Learning 
Objects http://www.rlo-cetl.ac.uk . Such evaluation has been widely deployed (Bath-Hextall et al,201 1). The 
dyslexic children were observed while using the prototype. To obtain more accurate data, they were also asked 
questions after they used the prototype. The whole situations were videotaped and replayed for further observation 
to confirm the answer. The results were presented in percentage. 

The study sample is the population of dyslexic students at all dyslexia schools that teaches mathematics in 
Klang Valley, Malaysia. The whole population consisted of thirteen students in the mathematic inclusion class. To 
address the ethical issue, the researcher has obtained permission from The Ministry of Education, The State 
Education Department, the school headmasters, as well as the teachers to conduct the test in classrooms. 

4. Development 

The first phase describes the analysis of content and user. Document analysis was performed where quiz, 
test, and report book have identified the pattern of majority difficulties faced by the dyslexic children on a particular 
topic: multiplication-of-two. As a continuation of the analysis process, teacher was interviewed and class 
observation was also made to confirm the situation. This identifies the need of designing IMLO on multiplication- 
of-two topic. 
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Figure 4: An evidence pertaining to the difficulty faced in answering multiplication-of-two questions. 

Students who face difficulty with multiplication of two are expected to find difficulty with other multiples as well. 

They might need to comprehend the concept in the beginning. 

It is understood that dyslexic children also have learning styles. Result of survey conducted identified that 
majority of dyslexic children were visual learners. However, the researcher suggested considering all learning styles 
for the strategy. This agrees with Reid (2005) who said that every effort should be made to organize the learning 
environment in a manner which can be adapted to suit a range of styles. 

The second phase describes the process of creating a storyboard with special instruction based on theories 
and expert opinions. It was suggested that the content be delivered using a story which relates to their 
environment/common activities. This approach will hopefully improve their understanding. The content is broken 
down into small steps containing specific topic. It is believed that dyslexic children can perform better when the 
content is small and specific. 

Preliminary studies were also conducted where Ronaldi et al (2011) identified the significance of using 
animation for the proposed special instruction. Animation using cartoon characters was used to describe certain 
complex information and concept such as simulation explaining two repeated addition concept, skip counting, 
numbers recalling (Figure 5). The animation created multisensory engagement through the use of graphic symbols 
motion pictures, sound and voice over. Reid (2005) stated that multisensory teaching approach incorporating visual, 
auditory, kinesthetic and tactile. Each of the visuals used were also carefully selected to help the dyslexic children 
retain the information through their imagination. 




Figure 5: Screen on the left shows animation of two candies to be put into three party packs representing repeated 
addition concept of 2 x 3. Screen in the middle shows animation of arrows, cartoon character (named Amir) reading 
the numbers and voice over for skip counting technique. Screen on the right shows animated cartoon characters 

recalling the skip counting concept. 
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The use of appropriate visuals was also considered. It was believed that relevant images referring to the 
dyslexic children environment help them to comprehend better. This statement was in agreement with Reid and 
Green (2007) who said adding pictures to the text aid comprehension. Furthermore, the cartoon character was also 
made to keep the attention and interest. The main character of Amir (Figure 5) was designed to appear at random 
throughout the learning process. His sudden appearances help to keep the attention and interest. Similar to Everatt 
(1999) opinion, the researcher believed that the dyslexic children need attention process while reading. 

The third phase describes the evaluation process. This phase focuses on evaluation of the IMLO based on 
feedback obtained from the dyslexic children by using the RLO Student Evaluation. Twenty one questions were 
selected to address certain aspects of the IMLO situations. These questions reflect the IMLO development stategy as 
described in the second phase. (Figure 6) 



RLO Student Evaluation 


Agree 


Neither 


Disagree 


Acceptance 








The content was appropriate and fitted my learning needs 


92.3% 


7.7% 


0% 


The RLO was pitched at the right level for me 


92.3% 


0% 


7.7% 


Attention 








The RLO encouraged me to reflect on the material 


100% 


0% 


0% 


1 liked the look and feel of the RLO 


100% 


0% 


0% 


The RLO was interesting and engaging 


100% 


0% 


0% 


The images and animations were valuable components of the RLO 


100% 


0% 


0% 


Usability 








The RLO was easy to use 


92.3% 


0% 


7.7% 


The on screen text was useful and helped me assess the amount of 
information each section contained 


76.9% 


0% 


23.1% 


Narrative content 








The narration made the RLO more engaging. I preferred this to text 
alone 


100% 


0% 


0% 


The RLO was well structured and easy to follow 


84.6% 


0% 


15.4% 


The RLO integrated well with the module and other teaching sessions 


100% 


0% 


0% 


Interactivity 


I enjoyed being able to work at my own pace 


92.3% 


0% 


7.7% 


The RLO was easy to navigate. I felt in control 


100% 


0% 


0% 


Activity 








The self-assessment helped me gauge how well 1 understood the 
material 


92.3% 


0% 


7.7% 


The activity was appropriate and aided my understanding 


92.3% 


0% 


7.7% 


Comprehension 








I am confident that I will be able to use the knowledge gained from this 
RLO in future practice 


92.3% 


7.7% 


0% 


1 like the idea that I can access this RLO whenever I need to 


100% 


0% 


0% 


I will use this RLO again 


92.3% 


7.7% 


0% 


The RLO will help me retain the information 


92.3% 


0% 


7.7% 


The RLO has aided my understanding and I feel I have achieved the 
learning objective 


100% 


0% 


0% 



Figure 6: Shows the percentage obtained based on the students agreement 

5. Results and Discussion 

IMLO was evaluated. All dyslexic children participated well and completed the user testing task. The 
answers were obtained using triangulation approach where feedbacks obtained from the questions were confirmed 
through observations. Overall, the results showed positive feedback 
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IMLO content was conceptualized through the story of a birthday party , where the narrative content was 
broken down into small steps. Majority of dyslexic children (92.3%) found such content appropriate and fitted their 
learning needs. Even though majority of them were visual learners, IMLO was designed for all learning styles. All 
of the dyslexic children agreed that the narration made IMLO more engaging and integrated well with the module. 

Amir cartoon character has been designed to engage the attention of the dyslexic children and due to his 
character, the children found the IMLO interesting and engaging. All of the dyslexic children liked the look and feel 
of the IMLO. They found that images and animation were valuable components of the IMLO. 

In terms of usability, 92.3 % of the dyslexic children found IMLO easy to use. 76.9 % of them also found 
that the on screen text was useful and helped them to assess the amount of information each section contained. The 
dyslexic children can easily explore the IMLO content as it was well structured and easy to follow (as agreed by 
84.6 % of them). They (92.3%) also enjoyed being able to work at their own pace. 



□ 92.3% Agree 
■ 7.7% Disagree 




Figure 7: Majority dyslexic children (97.3%) agreed that IMLO was easy to use. 

While using IMLO, the dyslexic children were engaged with activities. 92.3% of them found the activity 
was appropriate and aided their understanding. This helped them achieve the learning objective. IMLO was designed 
with specific visualization. The use of relevant images referring to the dyslexic children environment helped them to 
retain the information. The animation simulation explained the concept of repeated addition and boost their 
confidence in using the knowledge gained from IMLO in the future practice (92.3%). When they were asked 
whether they will use IMLO again, 92.3% agreed. IMLO hoped to be used as reusable Learning Object. 

6. Conclusion and Recommendation 

Understanding the dyslexic children and the need of learning a specific topic is important in developing 
IMLO. IMLO has been designed with specific strategies to suit to the dyslexic children’s needs. They need to relate 
the learning content to their environment and therefore narrative content was found useful. The choice of story helps 
to identify the visuals, graphics and icons that suit to their preferences. Using cartoon character design was also 
found helpful in engaging the dyslexic children’s attention. The researcher believes that digital/interactive comic can 
be employed to teach a certain topic. Through specific cartoon character design and narrative content, dyslexic 
children might have better reasons to focus and complete a task. The researcher is looking into that area. 

IMLO was successful in supporting the students understanding of multiplication-of-two. The IMLO was 
evaluated positively by the dyslexic children, and they reported the continued usage of the IMLO after the 
evaluation. IMLO managed to help students to understand difficult concept such as repeated addition and skip 
counting. The animation plays an important role to illustrate the concept and skills. Students are interested to plug 
gaps in their knowledge and keep their mind focused with the animation, graphics, and cartoon character displayed. 
The feedback showed that the IMLO supported learning as well as provided self control. The dyslexic children were 
able to perform the given task with minimum supervision. 
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Abstract 

This study examined the relationships between the structure of question prompts and the levels of critical 
thinking demonstrated by students’ responses in 27 online discussions. Discussion question types were classified 
using Andrews’ typology (1980): playground, brainstorm, focal, general invitation, lower-level divergent, 
analytical convergent, shotgun, funnel, and critical incident. Transcripts of students’ responses from twenty seven 
online discussions were analyzed for levels of critical thinking using Garrison’s four-stage Practical Inquiry Model. 
Differences in the impact on the levels of critical thinking were observed across the differing question types. Among 
the nine question types, critical incident and lower divergent questions were most effective in generating high levels 
of student thinking compared to other question types. Implications for instructors who are looking for general 
guidelines regarding how to structure their questions to elicit high quality responses are discussed. 

Introduction 

Online and blended forms of learning have grown considerably in higher education over the past decade 
and continue to grow at a rapid rate today. The growth of online and blended learning has prompted educators and 
researchers to identify “best practices” for the design of online learning experiences. Although a wide variety of 
instructional strategies can be used to encourage student learning online, discussion is one of the most commonly 
used pedagogical methods in online education since it has the potential for promoting critical thinking (Richardson 
& Ice, 2010; Walker, 2004; Yang, 2002) and knowledge construction (Pena-Shaff & Nicholas, 2004). 

Although online discussions have the potential to promote critical thinking, simply giving students the 
opportunity to interact does not automatically lead to meaningful discourse or the use of critical thinking skills 
(McLoughlin & Mynard, 2009). Based on the results of their research. Garrison, Anderson, and Archer (2001) noted 
that over 80% of students’ discussion posts reflected lower levels of thinking. Similarly, Gilbert and Dabbagh 
(2005) reported that approximately 75%-80% of their students’ online postings were at the lower levels of Bloom’s 
taxonomy (e.g., knowledge, comprehension, application). Gibson (1999) suggests that questions structured and 
constructed by instructors is the most basic tool of discussions that stimulate students’ thinking to construct 
understanding of the specific content. 

Over the years, research has consistently demonstrated a strong relationship between teachers’ questions 
and subsequent student responses (Bloom, 1956; Dillon, 1994). Much research has been done concerning questions 
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and question types in face to face environments and traditional classroom setting, but only a handful of studies have 
been conducted to explore the role of questions in online discussions, specifically, initial question prompts. 

However, as the popularity of online instruction continues to grow (Allen & Seaman, 2008), it is important to 
examine the nature of this relationship in the online environment as well. This study aims to fill this gap in the 
literature on online education. 

Purpose of the Study 

The purpose of this study is to examine the relationship between the structure of online discussion 
prompts/questions types and the quality of students’ responses, specifically in terms of levels of critical thinking in 
online discussions. The research questions guiding this study included: 

1. What is the relationship between the type of question prompt (Andrews typology) and students’ 
responses in terms of levels of critical thinking? 

2. Which question prompt types promote the highest levels of critical thinking? 

Theoretical Framework 

Garrison, Anderson, and Archer’s (2001) Community of Inquiry (Col) and Bloom’s six levels of cognitive 
processing was used as the theoretical framework to guide the study. The Col framework has frequently been used 
to assess critical thinking skills of students in online discussions (Richardson & Ice, 2009). 

Garrison et al. has developed Col model to explain the concept of critical thinking in online collaborated 
learning environments, through cognitive presence, social presence, and teaching presence. Cognitive presence is 
defined here as “the extent to which learners are able to construct and confirm meaning through sustained discourse 
in the critical community of inquiry” (Garrison et al., 2001, p.l). The Col reflects the critical thinking process of its 
learners and the means to create cognitive presence in the context of group collaboration. Col is grounded in a 
practical inquiry model (PIM) that provides a framework to assess the levels of CT in students’ online discussions 
postings. The PIM is based on the four phases of cognitive presence: (1) Triggering- become aware of a problem; 

(2) Exploration- explore a problem by searching/offering information; (3) Integration- interpretations/construction of 
possible solution, and (4) Resolution- providing potential solutions. These four phases are not considered discrete or 
linear since students may need to return to a previous phase (Swan, Garrison, & Richardson, 2009). According to 
Garrison et al. (2001), the PIM focuses on thinking processes and can be used as a tool to assess critical discourse 
and higher-order thinking in online discussions. Therefore, this framework was selected to examine the influence of 
the initial question prompts on the critical thinking. 



Methods 



Context 

The exploratory study was designed to examine the relationship between the verbal structure of question 
prompts and the level of students’ responses in online discussions. We examined discussion prompts from 10 
asynchronous courses, taught by seven different instructors during five semesters: spring and fall, 2008; and spring, 
summer, and fall, 2009. Three courses were taught primarily online while seven used online discussions to augment 
regular class meetings. Courses ranged in size from 9-221 students (n = 569) and represented six disciplines 
including Educational Technology; Educational Psychology; English Education; Literacy and Language; Speech, 
Language, and Hearing Sciences, and Veterinary Medicine. The students in each course engaged in online 
discussions related to course content during 16-week semesters. In general, students received participation points for 
the responses posted in the online discussions. 
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Table 3. 



Course and Participant Details 



N 


Discipline 


Course 


Level 


Semester 


Approach 


17 


Educational 

Psychology 


Advanced Educational 
Psychology 


Graduate 


Fall 09 


Blended 


9 


Educational 

Technology 


Educational Applications of 
Hypermedia 


Graduate 


Fall 09 


Blended 


29 


Educational 

Technology 


Educational Technology for 
Teaching and Learning 


Graduate 


Sum 09 


Web-based 


9 


Educational 

Technology 


Foundations of Distance 
Education 


Graduate 


Fall 08 


Web-based 


221 


Educational 

Technology 


Introduction to Educational 
Technology 


Undergrad 


Spring 08 


Blended 


178 


Educational 

Technology 


Introduction to Educational 
Technology 


Undergrad 


Fall 08 


Blended 


21 


English Education 


Composition for English 
Teachers 


Undergrad 


Spring 09 


Blended 


10 


Language and 
Literacy 


English Language Development 


Graduate 


Fall 09 


Blended 


62 


Speech, Language, 
& Hearing 
Sciences 


Introduction to Aural 
Rehabilitation Across the 
Lifespan 


Undergrad 


Spring 09 


Blended 


13 


Veterinary 

Medicine 


Management Topics for 
Veterinary Technicians 


Undergrad 


Spring 09 


Web-based 



Data Collection 

Discussion questions (n=92) collected from the 10 courses were classified into nine question types: 
playground, brainstorm, focal, general invitation, lower-level divergent, analytical convergent, multiple consistent, 
shotgun, and critical incident (See Table 1). To ensure accuracy in the categorization, all 92 questions were 
classified by two researchers independently and then compared for any differences. After reviewing each of the 
classifications, researchers discussed their differences and clarified individual interpretations in order to reach 
consensus. Eight questions were based on Andrews (1980) and one additional question “critical incident” was added 
to represent another category of question type by the researchers. 

After coming to consensus on the classifications for the 92 discussion prompts, we then selected 27 
discussions (3 from each of the final 9 categories) to use for the analysis of students’ postings. Question selection 
with associated discussion was based on the following criteria: questions representing different courses and 
disciplines, questions that best fit the description of the question type, and discussions with more interaction (depth) 
and posts (breadth). 
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Table 1 

Types of discussion prompts and their descriptions 



Type of 

Question 

Prompt 

1 Playground 



2 Brainstorm 

3 Focal 



4 General 
Invitation 

5 Lower 
Divergent 

6 Analytical 
Convergent 

7 Shotgun 

8 Funnel 

9 Critical 
Incident 



Description 



These questions represent the interpretation or analysis of a specific aspect of the material 
“playground” for discussion. Students have freedom of choice to discover and interpret the 
material. 

These questions represent generation of all conceivable ideas, viewpoints or solutions within an 
articulated issue. Students are free to generate any or all ideas on a given topic. 

These questions represent an issue and require students to make decision or take a position and 
justify it. Students can choose from limited or unlimited number of positions to support their 
viewpoints. 

These questions represent a wide range of responses in a formless or unfocussed discussion. 
Students are encouraged to give a wide range of responses within a broad topic with insufficient 
direction. 

These questions represent the simple reproduction of knowledge from comprehension with a lack 
of content richness. Students are required to analyze information/ material to discover the 
reasons, draw conclusions, or generalizations. 

These questions represent analytical thoughts for single correct answer. Students are required to 
examine a relevant material and produce a straightforward conclusion, summarize material, or 
describe a sequence of steps in a process. 

These questions represent multiple question-sentences and may contain two or more content 
areas. Students are expected to answer at least one fragment of the question. 

These questions begin with a broad opening question, follow with narrower question, and 
eventually funnel down to a very specific concrete question. 

These questions represent a scenario and ask students to respond using information from the 
material or experience. 



Data Analysis 

After identifying 27 discussions for coding, researchers independently coded students’ postings in four of 
the discussions using the Practical Inquiry Model (Garrison et al., 2001). Postings were scored at the message level, 
which varied in length from a sentence to several paragraphs. After coming to consensus on the codes for the 
responses in these four discussions, each researcher independently coded remaining 23 discussions. Then 
frequencies and percents of levels of CT across 9 question types were compared to explore which type has the most 
productive response in terms of levels of CT. Following this, discussion codes were entered into NVivo, a 
qualitative analysis software package. Matrix coding queries were then performed in order to examine relationships 
among specific, selected variables (question type, question level, etc.). 

Results and Discussion 

Question 1: Relationship between Question Type and Level of Response 

Results based on PIM (Table 2) shows that responses to Critical Incident (34%) and a small percentage of 
responses to Playground (6%), Analytical Convergent (5%), Funnel (4%), and Lower Divergent (4%) questions 
reached the highest level of CT at the resolution level. This was followed by Lower Divergent (69%), Shotgun 
(66%), Focal (55%), and Analytical Convergent (55%) question responses that mainly resulted in the second highest 
level of CT at the integration level. Although, responses to General Invitation (61%) and Brainstorm (56%) 
questions resulted in the greatest percentage of student responses at the exploration level, a small percentage of 
student responses to General Invitation (21%) questions also resulted at the lowest level of CT and were at the 
triggering level. 
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Table 2 



Percentage of Critical thinking levels expressed by students for different question prompts 





Critical 

Incident 


Play- 

ground 


Funnel 


Shotgun 


Brainstorm 


General 

Invitation 


Lower 

Divergent 


Focal 


Analytical 

Convergent 


Triggering 


6% 


5% 


3% 


4% 


6% 


21% 


5% 


4% 


17% 


Exploration 


31% 


41% 


44% 


29% 


56% 


61% 


22% 


39% 


13% 


Integration 


28% 


47% 


48% 


66% 


38% 


17% 


69% 


55% 


55% 


Resolution 


34% 


6% 


4% 


1% 


0% 


0% 


4% 


1% 


5% 



Question 2: Which questions prompt promote the highest levels of critical thinking? 

Overall, Critical Incident — that required students to respond to a scenario to create a solution — seemed to 
be the most influential in generating high levels of student thinking compared to other question responses (see Fig. 

1). These question types facilitated student responses that reached the resolution level (0=34%) and also at the 
integration (0=28%) levels of thinking. A review of students’ postings revealed that for Critical Incident question 
prompts, student tended to use their knowledge from the course material and personal experience to create a solution 
and give examples of how they can apply their solutions in a real life situation. Moreover, students justified their 
solutions with their personal examples and other resources, when asked to clarify any misunderstandings or 
conceptual conflicts of their peers. This suggests that structuring of questions that require original evaluative 
thinking to solve real-life problems and produce a solution can help students to think in more complex ways to 
generate a unique response. 



PIM Response Levels Generated by Question Types 
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Figure 1. Level of students’ responses when presented with different types of question prompts. 



Lower-Divergent, Focal, Analytical Convergent and Shotgun question prompts mostly fell into the 
“integration phase”. For instance, in lower-divergent question responses, students analyzed material to discover 
reasons to support opinions, they took a position and defended for responses of Focal question, and mainly 
summarize the required material during Analytical-Convergent question responses. These question prompts seemed 
to facilitate students synthesizing material and connecting relevant ideas from the discussion and describing the 
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issues presented in the discussion. Students also tended to negotiate meaning by producing well conceived thoughts 
within their responses as well as agreeing with their peers in order to reach a tentative solution. This finding is 
aligned with Zsohar & Smith’s (2008) conclusion that discussion prompts that incorporate course material, require 
reflective thinking to go beyond facts and use judgment to produce knowledge can facilitate higher level of critical 
thinking. 

General Invitation and Brainstorm questions mostly resulted in students achieving the exploration critical 
thinking phase. These questions allowed students to give a wide range of responses on a given topic. As compared to 
other questions, these questions did not facilitate higher levels of critical thinking. Due to the structure of these types 
of question (i.e., asking students to give a wide range of responses on a given topic), students tended to exchange 
ideas, search for explanations, and mainly use their personal opinions to support their arguments. This suggests that 
these questions are primarily effective in prompting students to share their initial ideas on a topic and thus, 
demonstrate their basic understanding of an issue. 



Limitations 

This study comprised an exploratory descriptive study; as such the results are not readily generalizable. 
When interpreting our results, it is important to recognize our study’s limitations. For example, the discussion 
questions were collected from 10 different courses, representing 6 different disciplines, including both 
undergraduate and graduate levels. Although others (Gilbert & Dabbagh, 2005; Schrire, 2006) have reported that 
graduate students tend to demonstrate a higher frequency of high-level responses than undergraduates, we did not 
examine differences among these populations. Future research might examine more closely the differences among 
students’ responses at different educational levels, as well as in different courses and disciplines, and with different 
instructors. This has the potential to lead to more specific guidelines for the types and levels of questions to use with 
different populations. 



Conclusions and Implications 

Although questions are used for many instructional reasons such as focusing attention, promoting recall, 
and encouraging reflection, using questions to stimulate critical or higher-order thinking is one of the most important 
goals of education (Gibson, 1999). Studies have shown that online discussions can support higher-order thinking 
(Gilbert & Dabbagh, 2005; Richardson & Ice, 2010), particularly through the use of effective questioning 
techniques. The results of this study provide additional evidence that initial question prompt can impact the level of 
critical thinking students achieve in online discussions. If the goal of online discussion is to facilitate higher levels of 
critical thinking, then the Critical Incident questions types can work the best. 

While encouraging higher levels of critical thinking is often the goal of online discussions, every discussion 
forum has a different objective; discussion questions should be designed based on the desired educational objectives. 
For example, if goal is to exchange ideas, provide explanations, and give suggestion for exploring relevant solutions, 
then Focal and Analytical Convergent question prompts be the best strategy. If the goal is to share ideas and develop 
a basic or foundational understanding of the issue, general invitation and brainstorm questions can be a possible 
choice. In this study. Brainstorm and General Invitation questions generated the lowest levels of thinking. Yet, 
instructors can modify these types of questions to target the higher or middle levels of CT. On the other hand, 
instructors should avoid using questions that are convergent and promote low-level responses. For example, Shotgun 
and Funnel questions that target a knowledge level response might not generate thoughtful or interactive 
discussions. Instructors can stimulate deeper thinking by omitting recall or memory level questions and using a 
Shotgun or Funnel question that requires students to describe underlying relationships or to make connections 
among ideas. 

The results of this study have important implications for instructors who teach online, especially those 
looking for general guidelines regarding how to structure discussion prompts to elicit high quality student responses. 
Because instructors have a lot of control over which questions they ask, and how they structure them, deliberate use 
of different types/levels of questions may enable them to engender higher quality responses from students. It is our 
hope that by examining the patterns observed in our results others will be able to modify their own discussion 
prompts to stimulate higher levels of thinking among their students. 
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Abstract 

This study investigated pre-service teachers ’ views and factors that influence their intentions to integrate Web 2.0 
technologies in their future classrooms. A mixed methods research design was used; qualitative interview data were used 
to triangulate quantitative survey data. Data were analyzed using the Decomposed theory of planned behavior (DTPB) as 
the theoretical framework (Taylor & Todd, 1995). Results suggest that pre-service teachers ’ attitude and their perceived 
usefulness of Web 2.0 technologies are strong indicators of their intention to use Web 2.0 tools. Additional findings 
indicate that pre-service teachers intend to use blogs, wikis, and social networking tools in their future classrooms. This 
study has implications for teacher educators who are preparing pre-service teachers to use Web 2. 0 technologies in their 
classrooms. 



Introduction 

The role of Web 2.0 technologies (wikis, blogs, social networking, etc.) in education is becoming increasingly 
prominent, both because of the need for students to develop 21 s1 century skills that will prepare them for living in an 
information society as well as their potential value of these tools for teaching and learning. Researchers have outlined 
potential benefits of using Web 2.0 tools in education including effective collaboration, active engagement, interaction, 
and communication (Dohn, 2009; Hartshome & Ajjan, 2009; Nelson, C hristopher. Mims, 2009; Shihab, 2008). However, 
the successful integration of these technologies depends on teachers’ abilities to create socially active learning 
environments that encourage cooperative interaction, collaborative learning, and group work (Nelson, et al., 2009; 
Coutinho, 2008). Students entering the field of education today, also called digital natives, tend to be savvy with social and 
communications technologies (Ob linger & Oblinger 2005). Yet, research shows that while pre-service teachers have 
positive attitudes and express their intentions to use Web 2.0 technologies, they are not prepared to use them in future 
classrooms (Lei, 2009; Gill & Dalgarn, 2008). The National Educational Technology Standards for teachers (NETS-T; 
International Society for Technology in Education, 2008) identify the importance to pre-service teachers of the 
fundamental knowledge, skills and attitudes for incorporating contemporary tools and resources to maximize student 
learning in education (ISTE, 2008). 

To prepare pre-service teachers to use Web 2.0 technologies in their classrooms it is important to understand the 
factors that impact their intentions to use these emerging technologies. Existing literature shows that while few studies 
have explored the factors influencing the technology integration efforts of pre-seivice teachers, rarely has research 
examined the potential factors that determine of pre-service teachers’ intentions to use Web 2.0 technologies in schools. 
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Purpose of Study 



The purpose of this study was two-fold. The first was to examine factors influencing pre-service teachers’ 
intentions to use Web 2.0 technologies in their future classrooms. The second was to explore pre-service teachers’ 
perceptions of the pedagogical benefits of using Web 2.0 applications to supplement their students’ classroom 
learning. The research questions for this study included: What factors best predict pre-service teachers’ decisions to 
use Web 2.0 technologies to supplement classroom instruction? What are pre-service teachers’ views of the 
pedagogical benefits of using Web 2.0 technologies to supplement their future classroom instruction? How does the 
follow-up qualitative data help explain the quantitative results? 



Theoretical Framework 



This study used the decomposed theory of planned behavior (DTPB) (see Fig. 1) as its theoretical framework 
(Taylor & Todd, 1995). The DTPB extends Ajzen’s (1991 ) theory of planned behavior (TPB), which focuses on the 
formulation of an intention to behave in a particular way. TPB suggests that a combination of behavioral intention and 
perceived behavioral control determines one’s actions. The DTPB explores subjective norms and perceived behavioral 
control more completely by decomposing attitude, subjective norms, and perceived behavioral control into lower-level 
belief constructs and states that behavioral intention determines behavior and that attitude, subjective norms, and perceived 
behavioral control are direct determinants of behavioral intention. The DTPB provides a comprehensive way to 
understand how an individual’s attitude, subjective norms and perceived behavioral control can influence his or her 
intention to use Web 2.0 (Ajjan & Hartshone, 2008). Moreover, it helps examine the relationship of factors that impact the 
adoption and use of new technologies more specifically (Taylor and Todd, 1995). 



r 




Fig. 1. Research framework for pre-service teachers’ use of Web 2.0 technologies based on decomposed theory of 
planned behavior (Taylor & Todd, 1995) 
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Methods 



A mixed methods research design was used to examine students’ intentions to use Web 2.0 technologies in 
their future classrooms. Specifically, this study used the convergence triangular mixed methods design in which 
different but complementary data were collected to validate and expand quantitative results with qualitative data 
(Creswell & Clark, 2007). Quantitative data were collected from an online post survey and qualitative data was 
obtained from follow-up semi-structured interviews to triangulate against the data obtained from the survey. 

Context 

The required course, composed of a 1-hour large lecture and a 2 -hour lab, helps pre-service teachers learn 
how to integrate technology tools within their future classrooms. As part of the course, students worked on a five- 
week project about educational uses of specific Web 2.0 applications (e.g., Facebook, PBWiki, Wordpress, etc.). 
Students worked in teams of 6-8 students to explore the assigned Web 2.0 technology and create instructional 
materials on how it could be utilized within a classroom environment. The finished project consisted of a 
collaboratively written wiki chapter of a Web 2.0 application including examples of its use, training materials on 
how to use it, as well as educational materials to inform others of its potential. At the end of the project, teams 
publically presented their Web 2.0 applications and demonstrated how it can be used in the classroom. 

Participants 

The study took place at a Midwestern university in spring 2010 semester. In total, 286 pre-service teachers 
completed the online survey. Of the participants, 196 (68 %) were females and 90 (32%) were males. The majority 
(90%) of the participants were 16-21 years of age, 16 (6%) were between 22 and 27 years, and 1 1 (4%) were above 
28 years of age. The majority of the pre-service teachers (63.6%) rated themselves as being very comfortable with 
computers, 33.6% rated themselves as fairly comfortable, and 2.8% rated themselves as a little comfortable. 

For the qualitative interviews, a purposive sampling method was used to choose participants based on the 
survey responses. Criteria for selecting the participants were two fold: pre-service teachers representing different 
majors and representing different grade level interests. Seven participants- three males and four females who 
represented an equal number within each criterion were selected for final interviews. 



Research Instruments 



Survey Instrument. 

The survey instalment consisted of three sections and was partially adapted from previous studies (Hartshorne 
& Ajjan, 2009). The first section of the survey included seven multiple choice items to determine the general 
demographics of the participants including name, e-mail, gender, age, year in school, major, and perceived comfort 
level with computers. Section two included five statements about participants’ views and intentions to use Web 2.0 
tools that explored participant’s levels of proficiency with Web 2.0 applications, actual use, perceptions of 
pedagogical advantages, and the Web 2.0 technologies that they intended to use in their future classrooms along 
with one open-ended item asking their thoughts of suing Web 2.0 technologies within a classroom environment. The 
third section of the survey consisted of modified items of the DTPB scale with a series of 7-point Likert-scale 
(Strongly agree to strongly disagree) to examine factors that influence pre-service teachers intentions to utilize Web 
2.0 technologies in their future classroom. Items focused on actual Usage/behavior (I would have no difficulty 
explaining why Web 2.0 technologies may or may not be beneficial). Behavioral Intention (I plan to use Web 2.0 
technologies in my future classroom), perceived behavioral control (Using the Web 2.0 technologies is entirely 
within my control), attitude (Web 2.0 will be useful in my teaching), and subjective norms (My students will think it 
is important to use Web 2.0 technologies in my classroom). 

Interview. 

The interview questions were developed based on the Web 2.0 attitude and the DTPB constructs to further 
explore teachers’ survey results and gain additional insights. Sample questions included: What do you think of using 
Web 2.0 technologies within a classroom environment? Would you use Web 2.0 tools in your future classrooms? 
Why or why not? What is the most important factor that influences your decision to use Web 2.0 in your future 
classroom? etc. Each interview session lasted approximately fifteen to twenty minutes. 
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Data Collection and Analysis 



Quantitative data were collected from an online post survey made up of three sections: demographic data, Web 2.0 
attitude scale, and the DTPB scale. The survey was administered at the end of the Web 2.0 project to measure factors 
related to pre-service teachers’ behavioral intentions to use Web 2.0 technologies and perceptions of the benefits of using 
Web 2.0 technologies in their future classrooms. The survey took approximately 10 to 15 minutes to complete. 
Demographic and attitude scale data were analyzed with descriptive statistics. The DTPB results were analyzed using path 
analysis to determining the factors and to estimate the degree of the linkage between variables that determine intention to 
adopt Web 2.0 technologies. 

Qualitative data were collected from one open-ended survey question and follow-up semi-structured interview 
questions with seven students to triangulate against the data obtained from the survey and to better understand the 
participants’ responses. Participants were interviewed after the conclusion of the course Web 2.0 project. An informed 
consent statement approved by the Institutional Review Board (IRB) was used to get participants permission to participate. 
Qualitative data were content analyzed using Miles and Huberman’s (1994) three step data analysis procedure. First, the 
data were coded into conceptual chunks and grouped into categories. Then, the categories with similar meanings were 
combined in order to identify the relationships and the key themes for formulating the assertions. The belief statements 
were examined and grouped according to the Web 2.0 attitude and the DTPB constructs. The beliefs that appeared key to 
students’ views about Web 2.0 and decisions to adopt new technology were identified. 

Results 

Factors Influencing Intentions to Use Web 2.0 Applications 

The path analysis results show that the pre-service teachers’ perceived usefulness of Web 2.0 technologies is the 
strongest determinant of their attitudes (see Fig. 2). Attitude in turn has the strongest effect on their intentions to use Web 
2.0 technologies in their future classroom. Regression results confirmed each of the three factors, attitude, behavioral 
intention, and subjective norm, explained a significant variance (71.5%) in behavioral intention (adjusted R2). 




*p< 0.05. **/><0.01. ***/><0.001. 

Figure 2. Path analysis of factors that influence pre-service teachers’ intentions to use Web 2.0 technologies in the 
classroom 
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Attitude. The results indicated that the pre-service teachers’ attitudes and their perceptions of the usefulness 
of Web 2.0 tools are the strongest determinants of their intentions to use Web 2.0 technologies. Regression results 
confirmed each of the three factors, perceived usefulness, perceived ease of use, and perceived compatibility, and 
explained a significant variance (78.3%) in attitude (adjusted R 2 ). Interview data revealed that pre-service teachers 
perceived Web 2.0 tools to be useful for student engagement, motivation, collaboration, communication, varied 
learning experiences, and holding students’ interest. Four out of five participants mentioned that they will use Web 2.0 
in the classroom to impact student learning and this in turn can affect their grades. For example, one participant said, “I 
definitely think that if the students could be excited about using a new technology like this [Web 2.0], it could 
definitely help in motivation and if they are motivated they can definitely do better in class.” 

Subjective Norm. Regression results confirmed each of the three factors — superior, student, and peer, — 
explained a significant variance (68.3%) in the subjective norm (adjusted R 2 ). Student influence had the strongest 
influence on the subjective norm, which, in turn, had a strong influence on behavioral intention. From the interviews 
“students influence” also emerged as the biggest factor in determining whether or not the pre-service teachers intend to 
integrate Web 2.0 into their teaching. For instance, one participant stated, “If they [students] are having trouble with 
lecture and I integrate a web 2.0 application and they just catch on and the test grades are awesome, I will use it.” 

Perceived Behavioral Control. Regression results confirmed each of the three factors — facilitating resources 
conditions, facilitating technology conditions and self-efficacy — explained a significant variance (62.2%) in perceived 
behavioral control (adjusted R 2 ). All three factors were found to influence the perceptions of behavioral control, which 
also had an influence on behavioral intention of pre-service teachers to use Web 2.0 technologies, with self-efficacy 
having the strongest influence. The participants’ interviews revealed that while most of the participants felt very 
comfortable and confident using Web 2.0 technologies, they thought that it might be challenging to meaningfully 
integrate Web 2.0 within the curriculum. As one of the participants said “I might use a little bit more effort just because 
we have to come up with new ideas that aren’t already out there but I mean other than that it’s [Web 2.0 technologies] 
really easy to use.” 

Perceptions of use and benefits of Web 2.0 Technologies 

In terms of their current uses of web 2.0 technologies, 43% of the students reported that they use wikis, 34.3% 
use social networks, 33.6% use instant messaging, and 34% occasionally use video chat to supplement their in-class 
learning. Additionally, many reported that they don’t use blogs (41.6%) and social bookmarking (44.8%) technologies. 
However, most of the pre-service teachers planned to use wikis (70.6%), blogs (42.7%), video chat (33.2%), and social 
networking (24.5%) applications in their future classrooms to supplement their students’ learning (see Table 1). 

Table 1 



Pre-service teachers ’ perceptions of the pedagogical benefits of Web 2.0 applications 





Blogs 

(%) 


Wikis 

(%) 


Social 

Net- 

working 

(%) 


Social 

Book- 

marking 

(%) 


Instant 

Messaging 

/Chat 

(%) 


Video 

Chat 

(%) 


Video 

Sharing 

(%) 


Improve student- teacher interaction 


29.4 


13.6 


28.7 


1.7 


10.8 


11.5 


2.8 


Improve student learning 


9.8 


63.3 


6.3 


3.5 


1.0 


6.6 


7.7 


Improve student satisfaction with the 
course 


20.3 


24.1 


26.2 


6.6 


2.4 


4.5 


12.9 


Improve interaction with other 
students 


11.2 


5.6 


47.2 


4.5 


16.8 


11.2 


1.4 


Improve student grades 


13.6 


49.0 


10.8 


11.2 


3.5 


3.5 


3.8 


Improve student writing ability 


53.8 


23.1 


5.6 


4.2 


4.5 


.3 


3.1 


Easy to use/share content knowledge 


15.4 


38.1 


18.5 


9.1 


5.9 


4.9 


6.6 


Improve critical thinking with 
collaborative learning 


19.9 


37.8 


13.3 


7.0 


5.2 


9.1 


5.6 
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In terms of pre-service teachers’ perceptions of the benefits of using Web 2.0 technologies to supplement 
classroom instruction, the results showed that the pre-service teachers felt that the use of different Web 2.0 
technologies could provide students with several benefits. They viewed wikis as the most useful Web 2.0 application 
in terms of improving student learning (63%); improving student grades (49%), ease of use/sharing content 
knowledge (38%), and improving critical thinking with collaborative learning (38%). In terms of improving student 
writing ability (54%) and improving student-teacher interaction (29%), blogs were viewed as most beneficial. Social 
networking technologies were perceived to improve interaction with other students (47.2%), improve student- 
teacher interaction (28.7%), and improve student satisfaction with the course (26%). 

Interview and open-ended survey data also revealed pre-service teachers’ thoughts that Web 2.0 
technologies can offer many advantages to students learning through more interaction, communication, improved 
writing, and extended class time. For example, one of the pre-service teachers stated, “Web 2.0 technologies 
enhance learning by providing various learning opportunities for students. Students can use them to connect with 
peers and teachers outside the classroom and to bring other elements into the classroom (i.e., virtual field trips).” 
Another commented, “I think that blogs are a great way to communicate in the classroom and get students more 
involved and interacting with each other.” Similarly, another one said, “Wikis and blogs can be a great tool for 
compilation and organization of student writings as well as a place to receive feedback and improve upon work.” 



Discussion 

This study explored pre-service teachers’ views and factors that influenced their intentions to integrate Web 
2.0 technologies in their future classrooms. The results revealed that pre-service teachers had both positive attitudes 
and high intentions to integrate Web 2.0 technologies in their future classrooms as teachers. 

Factors Predicting Pre-Service Teachers Intentions to Use Web 2.0 Technologies 

The path analysis results show that the pre-service teachers’ perceived usefulness of Web 2.0 technologies 
is the strongest determinant of their attitudes. Attitude in turn has the strongest effect on their intentions to use Web 
2.0 technologies in their future classroom. Interview data supported this finding and revealed that pre-service 
teachers viewed Web 2.0 technologies to be useful to enhance their teaching as well as supplement their students’ 
in-class learning. This perception of the usefulness of Web 2.0 might be due to the pre-service teachers’ exposure to 
Web 2.0 technologies during the Web 2.0 project that helped them learn how to integrate Web 2.0 technologies 
within their future classrooms. This is consistent with Coutinho’s (2008) study that found that providing technology- 
rich experiences to pre-service teachers with Web 2.0 technologies has a positive influence on pre-service teachers’ 
intentions to use them in the classroom. 

In addition, all three groups — administrators, peers, and students — are the key determinants of subjective 
norms of pre-service teachers. Subjective norm in turn, did not influence behavioral intention. However, students 
exerted a stronger influence on subjective norms as compared to the other two groups. These results imply that pre- 
service teachers’ behavior is likely to be affected by their future students’ expectations regarding the use of 
technology. The interview data also suggested that pre-service teachers believed that integrating Web 2.0 
technologies will make learning more meaningful and relevant for their students because they are already using 
these technologies outside of the classroom. This finding is supported by Shihab (2008), who suggested that today’s 
students expect learning to take place using modern digital communication tools, therefore Web 
2.0technologiesshould be integrated into the classroom due to the technology expectations of digital natives as well 
as its great potential for teaching and learning. 

Self-efficacy and facilitative technology conditions were found to influence the perceived behavior control 
of pre-service teachers toward the intention to use Web 2.0 technologies. On the other hand, facilitative resource 
conditions did not influence the perception of behavioral control. However, self-efficacy is found to be the greatest 
determinant of pre-service teachers’ perceived behavior control. This is corroborated by the findings of previous 
studies, which showed computer self-efficacy to positively influence teachers’ view and intentions to use and 
integrate computers (Anderson & Maninger, 2007; Giallamas & Nikolopoulou, 2010). Although, pre-service 
teachers expressed high self-efficacy in using Web2.0 applications, their self-efficacy related to integrating Web 2.0 
applications in lessons within classrooms was low. This might be due to their lack of actual classroom experience. 
Coutinho (2004) suggested that teacher education programs can provide pre-service teachers more opportunities to 
reflect on the pedagogical uses and implications of Web 2.0 technology integration. 
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Perceptions of Use and Benefits of Web 2.0 Technologies 

The results of the study indicate that a majority of the participants intend to use Web 2.0 technologies in 
their future classrooms. They believe that integration of these technologies into the teaching and learning 
environment has the potential to improve student learning. This finding is comparable to other research findings that 
consider Web 2.0 technologies useful having great potential for teaching and learning in the classroom (Coutinho, 
2008; Hartshorne &Ajjan, 2009; Shihab, 2008). While pre-service teachers mentioned many benefits, they also 
noted that successful integration depends on how selected Web 2.0 technologies relate to the content being taught, 
learning goals, and age level of the students. This implies that in addition to the benefits of using Web 2.0 
technologies, pre-service teachers are aware of the context of meaningful integration into classrooms. 

Conclusions and Implications 

This study explored factors that affect pre-service teachers’ intentions to use Web 2.0 in future classrooms 
and found that pre-service teachers had both positive attitudes and high intentions to adopt Web 2.0 technologies in 
their future classrooms as teachers. 

The results of this study provide evidence that pre-service teachers’ attitudes and perceptions of usefulness 
of Web 2.0 tools are the strongest indicators of their intentions to use Web 2.0 tools. Behavioral intention, in turn, is 
a strong indicator of actual behavior. Thus, the focus of teacher technology programs preparing pre-service teachers’ 
to effectively integrate Web 2.0 in the classroom might be on improving their attitudes toward emerging 
technologies as well as enhancing their perceptions of usefulness of Web 2.0 use by providing appropriate training 
and knowledge of how they can impact student learning. Moreover, pre-service teachers should be given 
opportunities to practice using these technologies in actual classrooms during their student teaching experiences. 
According to Albion (2008), pre-service teachers should learn about Web 2.0 by immersing them in authentic 
practice. 

The results also indicate that most of the pre-service teachers intend to use blogs, wikis, and social 
networking tools in their future classrooms. They believe that integration of these technologies into the teaching and 
learning environment has the potential to improve student learning. While pre-service teachers believe that Web 2.0 
technologies have great potential in K-12 education, they also noted that it all depends on the teacher’s ability to 
meaningfully integrate these technologies with the content being taught, and age level of their students. Hence, it is 
important for teacher educators to help pre-service teachers understand meaningful connections between technology, 
content, and pedagogy. According to Lei (200 1 ), meaningful technology integration can happen by helping pre- 
service teachers develop technological pedagogical content knowledge (TPCK). Thus, teacher education programs 
should include strategies that pre-service teachers can use to affect their student learning, according to their grade 
level interests and specific subject areas. 

This study contributes to the field by revealing pre-service teachers’ attitudes and perceived usefulness as a 
significant predictor of their intentions to use Web 2.0 in their teaching. Future research can now determine the 
specific interventions that will help to increase pre-service teachers’ attitudes and perceived usefulness of Web 2.0 
technologies. Teacher education programs should focus on these factors to enhance teacher education programs and 
prepare pre-service teachers to able to use these tools effectively. Moreover, longitudinal studies may be designed to 
determine if these beliefs about using Web 2.0 technologies translate into actual use in the classroom during student 
teaching experiences. 
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Abstract 

Does learner participation vary depending on the learning context? Are there characteristic features of participation 
evident in formal, non-formal, and informal online learning environments? 

Six online learning environments were chosen as epitomes of formal, non-formal, and informal learning contexts 
and compared. Transcripts of online discussions were analyzed and compared employing Transcript Analysis Tools 
for measures of density, intensity, and reciprocity of participation (Fahy, Crawford, & Ally, 2001), and mean reply 
depth (Wiley, n.d.). This paper provides an initial description and comparison of participation patterns in a formal, 
non-formal, and informal learning environment, and discusses the significance of differences observed. 



The literature examining learning in online environments is extensive, but it is primarily focused on formal learning 
environments in higher education (cf. Anderson, 2003; Brooke & Oliver, 2006; Garrison, Anderson, & Archer, 

2003; Luppicini, 2007; Murphy & Coleman, 2004). Formal environments typically require learners to engage each 
other online in specific, externally defined ways, whereas non-formal environments impose fewer controls on 
learner activities, and informal environments impose even fewer still. The nearly exclusive attention to formal 
settings limits our understanding of how learners make use of other types of contexts in self-directed learning. One 
of the most significant challenges in educational technology is to understand how learners participate in non-formal 
and informal learning networks to construct their own learning spaces. 

By formal, we refer to educational contexts usually characterized by learners in classes being taught by teachers who 
deliver comprehensive, multi-year curricula, which is institutionally bound to a graduated system of certification 
(Coombs, 1985). In sharp contrast, informal education is often characterized as unorganized, unsystematic, and 
regularly serendipitous (Selman, Cooke, Selman, & Dampier, 1998). This type of learning is the lifelong process of 
learning by which people acquire and accumulate knowledge skills, attitudes and insights gathered from a lifetime of 
experiences. For the purposes of this research, we focus on a third category of education, non-formal learning, that 
straddles these two seemingly polar learning contexts. Selman, et.al. (1998) identify non-formal learning as that 
which “comprises all other organized, systematic educational activity which is carried out in society, whether 
offered by educational institutions or any other agency. It is aimed at facilitating selected types of learning on the 
part of particular sub-groups of the population” (p. 26). For example, non-formal education may include such 
activities as professional development interest groups or community education initiatives. These alternative group 
learning contexts are usually characterized by participants who share expertise and knowledge, and may or may not 
include a content expert. 

In informal and non-formal learning environments, learners need to exercise various degrees of self-directedness in 
their approaches to their learning. Some authors have characterized the self-directed learner as learning alone, 
whether under the tutelage of an instructor or agency, or completely independent of such structures (Tough, 1971; 
Selman, Cooke, Selman, & Dampier, 1998). Flowever, we expand the notion of independence to include being 
independent of the structural contexts of education; any particular learner or group of learners may manifest 
elements of self-directedness in their learning whether it be within a formal, non-formal, or informal learning 
environments. 
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Methods 



Six distinct environments were selected for analysis, including two formal, two non-formal, and two informal 
groups. We attempted to select typical groups, not representative groups, and our selections were deliberate but 
based on convenience. The formal environments included an online graduate level course that met for 26 weeks 
from September to April, and another that met for six weeks during an accelerated term, and that included 
prescribed online discussions weekly. The non-formal environments were run as classes that ran for 13 weeks, and 
participation in the online conversations was encouraged but entirely voluntary. The informal environments were 
selected from a wide array of possible groups, and the selection criteria required that in the environments: 

— active conversations were conducted for no fewer than 13 weeks 

— participation was open to anyone; 

— conversations were held in public; 

— participation was voluntary; 

— topics of conversation were created by participants; 

— participants were free to come and go. 

In every case, transcripts from the online discussion boards were analyzed. Because the informal environments we 
selected included many more threads of conversation than we needed for analysis, we randomly selected 25 
conversations by generating a table of random numbers (1 to 100) and drew the first 25 numbers associated with the 
most recent 100 conversation threads in each environment. 

Fahy, Crawford and Ally (2001) proposed several useful measures of describing interaction that they called 
collectively the Transcript Analysis Tool (TAT). The TAT includes methods of measuring density, intensity and 
persistence of interactions in transcripts of online discussions. We drew on their recommendations and elaborated 
some of them to analyze interactions in our data, particularly transcripts of asynchronous discussions. 

Density 

Fahy, Crawford and Ally's (2001) definition of density is "the ratio of the actual number of connections observed, to 
the total potential number of possible connections." It is calculated by using the following formula: Density = 
2a/N(N-l), where "a" is the number of observed interactions between participants, and "N" is the total number of 
participants. Density is a measure of how connected individuals are to others in a group, and the idea is that a higher 
degree of connection is a positive indicator of community. Fahy, Crawford and Ally (2001) caution that the measure 
of density is sensitive to the size of the network, so larger groups will likely exhibit lower density ratios than will 
smaller groups. 



Reciprocity 

A particularly important TAT measure for the purpose of understanding community is "S-R ratio", a formula to 
measure the parity of communication among participants. We referred to this as a measure of "reciprocity", and we 
felt that truly engaged groups who form communities will exhibit high degrees of reciprocity. 

For this analysis we only included interactions that were not directed to the group, but only those directed to 
individuals. So, for example, a general question tossed out to a group would not be counted, but a reply to the 
originator of a message would be counted. In every case, regardless of topic, the communication was directed to a 
particular person, instead of to the group or to nobody in particular. 

As an initial step in the analysis, we charted the number of messages sent and received among participants in the 
group. The S/R ratio (sent to received messages) is an indication of the reciprocity of messaging within the group. 
Ratios approaching 1.0 indicate a high degree of reciprocity. Ratios considerably higher or lower than 1.0 indicate 
disparity in the communication. High numbers indicate that the individual was communicating to others, but not 
receiving as many communications in return. A low number indicates that a higher number of messages were 
received than were sent in response. We speculate that a healthy, vibrant community exhibits a high amount of 
reciprocity among members of the group. 
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Intensity 

Fahy, Crawford and Ally recommended measures of intensity to determine whether participants were authentically 
engaged with each other, not merely carrying out their responsibilities in a course. They argue that it is a useful 
measure of involvement because it involves measures of persistence and dedication to being connected to others in 
the group. This measure has more meaning for analyzing interaction in formal settings, as 

One measure of intensity is "levels of participation, " or the degree to which the number of postings observed in a 
group exceeds the number of required postings. So for example, if a group of students is required to make a total of 
200 postings as part of the course requirements, and the group actually makes 600 posts, it would yield a 
participation ratio of 3.0. While this is a useful measure, the number of superfluous and thoughtless replies to 
postings inflates it. Also, it does not fit well in informal environments where there are no required postings. For 
purposes of comparing environments, we took the position that we would anticipate each participant to make 2 posts 
in each discussion as a base line, and anything above that number would contribute to the measure of intensity. This 
was not entirely arbitrary, given that this was the expectation set out in the formal environments. 

Another measure of intensity recommended by Fahy, Crawford and Ally (2001) is persistence, or the level to which 
participants pursue topics. A measure of persistence is intuitively appealing, as it addresses how deep are the 
discussions we have observed in formal, non-formal, and informal environments. Even if there are fewer 
participants in one environment than another, are there differences in how persistent the conversations, and how 
replies ladder inward as individuals follow a thread of conversation deeper? In order to get at these questions, we 
turned to a measure of mean reply depth (MRD) proposed by Wiley (undated) in a working draft of a paper he 
prepared on the topic several years ago. 



Wiley’s approach proposes that replies in threaded conversations indicate that discussions are happening, and that 
levels of replies (reply depth) are positively correlated with the depth of discussion. Wiley proposed a formula that 
assigns increasing value to each level of reply in a conversation that can be used to calculate a mean reply depth for 
any thread in a conversation. 

r\ 

— — =9bu'ota 

n 

In this formula, A crude is an uncorrected measure of mean reply depth (MRD). In the formula, r is the reply depth of 
the ; ,h message, and n is the total number of messages in the group. Each level of reply is given a value depending 
on its depth, with top level messages valued at 0, first level replies given a value of 1, second level replies given a 
value of 2, and so on. 

In order to correct for the confusion that can obtain from comparing conversations that have top level messages and 
no replies with conversations that have the same total value but with more active participation, Wiley proposed an 
adjusted MRD formula that accounts for top level messages that have no replies (b), and corrects for them. 

q = qcinqe X ((U-p)\U) 

The MRD measure does have weaknesses that Wiley carefully notes. The MRD is a measure of activity, not quality 
of conversation. Each reply counts equally, whether it is a simple greeting or a deep critique. Also, participants in 
discussion groups do not always reply in correct thread positions; they sometimes reply higher or lower in the thread 
structure than intended. But with these cautions in mind, the MRD does provide a more robust and meaningful 
measure of the depth of activity and so provides a better indication of the intensity of a conversation. Also, the 
weaknesses can be mitigated by qualitative analysis of the original data. 

Following recommendations from Wiley (n.d.) we suggested the d values can be categorized as: 

< 0.3 = monologue, moribund 

0.3 — 1.2 = simple question and answer; chatting 

>1.2 = multilogue, discussion 
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Results 



As a preamble to considering our results, we acknowledge that learners participate in online learning environments 
for a host of reasons, responding to personal desires or needs and external requirements or pressures. Before 
offering a few generalizations about audience differences in the settings we observed, it is important to recognize 
that any environment includes individuals who are pursuing their own learning agendas, and any attempt to 
generalize will be filled with exceptions and flaws. Nevertheless, it seems reasonable to speculate that membership 
in formal learning communities is significantly influenced by program requirements and course designs, as well as 
by personal and professional compulsions. If a course is required as part of a credential, learners may have a deep 
interest in the broader area of study, but not necessarily the course under study. The challenge of building a strong 
sense of community in groups such as these is peculiar to their populations. 

We speculated that generally speaking, formal groups differ significantly from non-formal and informal learning 
environments, where participation is based on affinity rather than requirement or fiat. Online informal learning 
communities usually depend on the participation of relatively autonomous, independent individuals. In some non- 
formal and most informal online communities participants can engage or disengage from the group easily and 
without personal consequence, and they can sometimes participate in the community without revealing who they are 
to the other participants. 

Overview of Participation in Selected Environments 

First, we will summarize what we found when we compared participation in six selected environments — two each 
that were formal, non-formal, and informal according to our criteria. Table 1 summarizes the key observations. We 
then turn our attention to a closer look at mean reply depth within groups as an indication of the persistence and 
depth of the conversations. 
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Table 1. Summary of Participation Analyses Comparing Formal, Non-formal, and Informal Groups. 





Informal 1 


Informal 2 


Non-formal 

1 


Non-formal 

2 


Formal 

1 


Formal 

2 


Total msgs 


785 


320 


97 


175 


1041 


764 


# Discussions 


25 


21 


12 


6 


7 


19 


# participants 


506 


82 


12 


8 


18 


8 


#msgs/participa 

nt 


1.55 


3.90 


8.08 


21.88 


57.8 


95.5 


Msgs/discussion 


31.40 


15.24 


8.08 


29.17 


148.71 


40.21 


Density 


cliques 


cliques 


0.47 


.40 


1.0 


0.78 


Intensity 


.03 but 
variable 


.09 but 
variable 


0.34 


1.82 


4.1 


2.51 


Reciprocity 


low 


low 


0.92 

(sd=.94) 


1.74 

(sd=4.77) 


1.10 

(sd=.42) 


0.96 

(sd=.37) 


Grand MRD 


0.71 


1.16 


0.60 


0.70 


1.76 


1.40 



The first thing that jumped out at us was the difference in the number of messages and discussions. The number of 
discussions was prescribed for the most part in the non-formal and formal groups. While participants were permitted 
to start their own conversations, they seldom did. So, in these groups, the number of discussions was defined 
externally, at least mostly. By contrast, the members of the informal groups initiated all of their own discussions. 

We were forced to develop sampling criteria for the informal groups, because the number of discussions numbered 
in the hundreds or thousands, and we wanted to examine the groups in ways that were roughly equivalent. So the 
raw number of messages and discussions is accurate for the formal and non-formal groups, but are a small 
percentage of the actual number in informal groups. 

The actual number of messages was considerably fewer in the non-formal group, with higher message counts 
observed in informal and formal environments. But interestingly, when the number of participants was taken into 
consideration, we observed that there were dramatically fewer messages per participant in the informal group, with 
the number higher in non-formal and bursting in formal environments. Without question, this was related to the 
compulsory nature of participation in the formal groups, and probably the expectation of participation in the non- 
formal groups. 

Taking a closer look at the interaction patterns, we drew on Fahy, Crawford and Ally’s (2001) TAT indicators of 
intensity, density and reciprocity. Again, because the informal environments did not have a membership that could 
be tracked reliably, we adjusted our methods of observation, but in every case tried to inform our understanding of 
the feature under investigation. We suggest that as a result we can draw some interesting speculations from our 
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observations, but we do not make any claims about the reliability of comparisons with the informal groups. We were 
able to employ comparative measures between formal and non-formal groups, but the unstable membership in the 
informal environment would not permit us to use the same assumptions when we applied the TAT measures. As 
Reeves (2011) would probably observe, we were trying to achieve a balance between rigor and relevance, with a 
preference for relevance. 

Given these cautions, we found dramatic differences among the three environments. We defined intensity as the 
ratio of the number of postings that exceed expectations to the number of expected postings. In the formal group, 
this was counted as the ratio of postings that exceeded the requirements in the class (two per discussion topic). In 
order to keep the comparisons parallel, we set the same level of expectation as the baseline in the non-formal and the 
informal groups. In this case, we saw that the intensity of discussions in the formal group was considerably larger 
than in the non-formal group, and in fact, the non-formal group fell well below minimal expectations in one case 
(formal intensity = 4.1 and 2.51; non-formal intensity = .34 and 1.82). The informal group, on the other hand, 
repeatedly demonstrated high intensity on several discussion threads, and almost no intensity on some, and yielding 
composite intensity ratios of .03 and .09. Interestingly, the comparison of intensity in all of the groups demonstrated 
dramatic differences within environment types. While we don’t want to over-interpret this finding, it might suggest 
that the designs of the experiences and the topics under consideration may have contributed to the intensity exhibited 
by participants. But overall, the grand measure of intensity was higher in formal and non-formal groups than in the 
informal groups we observed in this case. Again, we can’t draw conclusions from the observation, but it allows us 
to generate some interesting hypotheses. 

A measure of density is a ratio of the number of actual connections to the number of potential connections among 
participants. Density asks whether all possible connections among participants are being made; in other words, does 
everyone in the network connect with everyone else? We found that a greater number of people in the formal 
environment connected with fellow participants than in the non-formal environment (formal density = 1.00; .78; 
non-formal density = .47; .40), but this was at least partly an artifact of the measure of intensity. Fewer people were 
engaged in the first place, so fewer connected with each other. While not surprising, it is another indication that the 
community bonds in the non-formal group were weaker than in the formal group. Once again, the informal group 
was curious. Because people came and went in the group more casually, it was difficult to track density in the same 
way. But we did find that density was lower in informal environments. In discussion threads in the informal 
environment, there was clustering around the person who began the conversation, but fewer connections among 
individuals responding. Conversations were largely bi-directional, not multi-directional. 

Reciprocity among participants is a measure of the ratio between the number of messages received by individuals to 
the number sent. In other words, did people realize balanced conversations in the group, which would be represented 
by a ratio of 1 .0 if individuals received and sent the same number of messages. In this case, we found that the mean 
reciprocity of participants in formal and non-formal environments was high and similar with one exception (formal 
reciprocity = 1.10; .96; non-formal reciprocity = .92; 1.74). The mean reciprocity of the groups masked considerable 
differences. We found that the standard deviation for the formal group was comparatively low (s.d. =.42; .37) 
indicating that reciprocity did not vary across individuals in the formal group as much as it did for individuals in the 
non-formal group (s.d. = .94; 4.77). There was one outlier in the non-formal group who skewed the measure of 
reciprocity and also accounted for the unusually high standard deviation. When this person’s data were removed 
from the analysis, the reciprocity was .89 with a standard deviation of 1.4. Once again, the informal group 
demonstrated a considerable amount of variance, with very low reciprocity for the group, but this was expected, 
given the voluntary, occasional and casual nature of interaction in this environment. Yet as a casual anecdotal 
observation, we noticed people were considerate of each other in the group; when somebody posted a comment, the 
person who posted the original topic was often attentive and responsive. We saw incidents of good manners, if not 
community. 



Mean Reply Depth 

Initially, we compared the grand mean reply depth for each environment, which was calculated as the average MRD 
for all of the conversations in the group (see Table 2). 
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Table 2. Comparison of Grand MRD for All Groups 





Informal 


Non-formal 


Formal 


Group 1 


.71 


.60 


1.76 


Group 2 


1.16 


.70 


1.40 



The Grand MRD revealed that that non-formal groups exhibited the lowest Grand MRD, with informal groups 
somewhat higher, and formal groups considerably higher than the others. But the grand MRD for informal groups 
varied considerably, which suggests that the environments did not behave consistently. Clearly, the Grand MRD 
masked other factors that were at work. 

When we compared MRD patterns in formal, non-formal and informal settings over time, considerable differences 
and patterns emerged. Figure one offers a comparison of the MRD for discussions held over time for fonnal and 
non-formal groups, and 25 randomly selected discussion topics from the most recent 100 for the informal groups. 



Figure 1. Comparison of MRD over time in informal, non-formal, and formal groups (bold line indicates Wiley’s 
(n.d.) interpretation of when the MRD level reaches multilogue or discussion). 
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In formal environments, participation was initially high, and while conversations fluctuated, they generally grew 
over time as participants moved beyond assigned postings and added their own contributions voluntarily, and the 
MRD reflected this pattern. In almost every discussion, the MRD exceeded Wiley’s (n.d.) threshold to be considered 
a discussion. As an aside, these observations were made of two selected environments. One other formal 
environment we observed followed a similar pattern as these, but returned a lower MRD overall, seldom exceeding 
the threshold for discussion (see figure 2). We think this indicates that there is a continuum of depth of conversation 
that can happen in any environment, and we speculate that formal environments may be particularly vulnerable to 
differences. Because formal environments require a minimal amount of discussion, it is possible that unobserved 
factors in the context of the class (e.g., content, general affect, instructor presence) may serve to encourage or 
discourage deeper levels of commitment and conversation. 

Figure 2. MRD graph of a formal environment with lower participation and depth. 




In the non-formal environment we observed, participants were encouraged but not required to participate. In these 
cases, initial participation rates were not as high as in formal environments, as a few participants chose not to post to 
the discussion board. In addition, we repeatedly observed that participation fell off steeply and quickly as the course 
neared its end, and this was mirrored by the MRD pattern. The MRD hovered at the level of chat, and never 
exceeded the threshold to be considered a discussion. 

In the informal environment, where participation was entirely voluntary, a completely different pattern emerged, one 
that can be described as effervescent. Participation rose and fell over time, apparently according to the amount of 
interest generated on a particular topic. Some topics drew audiences; others remained relatively quiet. But as a 
result, it was apparent that participation patterns were mediated by the personal interest of participants in topics, 
rather than by fiat (as in the formal learning environment) or by duty (as in the non-formal environment). Once 
again, the MRD pattern followed the participation pattern. Approximately one-third of the conversations could be 
classified as discussions, while most were categorized as chat or moribund. A review of the topics that drew higher 
rates of participation revealed that they might be provocative, humorous, profound, or personal, but in every case 
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they invited conversation. So in the case of our informal learning environment, participation seemed to be less 
about nurturing the group, and more about nourishing the group - offering the audience something that drew them 
into a conversation. And the audience judged what was worthwhile and what was not. 

In every learning environment we observed, except in non-formal environments, there were bursts of engagement in 
online discussions where participation was high and deeply engaged. This caused our research team to coin the label 
“principle of intensity” to describe what we thought was at the heart of the spikes of participation we observed. We 
speculated that intensity might be motivated by a number of catalysts in learning environments: social advocacy, joyful 
learning, emotional connections to ideas, and even associations with someone who is important or provocative. But in 
online learning, content also seemed to be an essential ingredient for intensity that was present, regardless of the catalyst. 
In other words, the interactions were about something significant that was shared by the group, a feature that has been 
labeled “object-centered sociality” elsewhere (Zengestrom, 2005). When individual learning is about something 
meaningful to members of the group, intensity can ignite, and it can appear in both synchronous and asynchronous 
discussions. 

We also speculate that the design and implementation of the learning environment is also complicit in the patterns of 
participation we observed. The measures of intensity in the formal environments, while limited in their explanatory value, 
were considerably higher than the measures of intensity in the informal and formal environments. Individuals, when 
required to participate, engaged in a higher amount of additional and extra-curricular conversation. When initial 
engagement was only recommended or left completely to the discretion of participants, there was little in the way of 
engagement beyond modest expectations. This begs the question of whether forced participation contributes to the 
development of more casual kinds of connections among participants. 

It also appears that the design of discussions in learning environments may influence the depth of the discussion that takes 
place. This is intuitively pleasing. If instructional leaders offer well-designed topics for conversation in formal 
environments where expectations for participation are high, then conversations may be deeper and more persistent. 

Recommendations for Research 

As tantalizing as these preliminary findings may be, they only scratch the surface of what might actually be happening 
when individuals participate in intentional learning activities in online learning environments. We realize that the meaning 
of these patterns is embedded in the conversations themselves — in their content, tone, and in the intentions of participants. 

The study of participation patterns is valuable in pointing out directions for additional investigations of the conversations. 
In addition to studying these patterns, our research team has been coding transcripts of the discussions to extract themes in 
the conversations we can map against participation to understand the meaning of the interactions we have observed. 
Findings from those analyses will be reported in subsequent papers. We are also interested in learning more about the 
effect of the design of environments on participation, and particularly the quality of participation in those environments. 

We also suggest that other aspects of conversations deserve attention. One area that interests us is how individuals initiate 
and elicit participation from others. In other words, what kinds of interactions invite discussion? This paper was based 
exclusively on asynchr onous discussions on discussion boards where much of the discussion is in response to a posed 
question or idea. Synchronous conversations in chats, videoconferences, messaging systems and the like also offer unique 
ways to look at how groups form and leam together. Of course, much of that kind of conversation is challenging to code 
and analyze. 

We suggest that conversation analysis may be a useful tool for conducting analyses of the dynamics of conversation, 
and especially helping us understand what kinds of communication attracts participation from others. Using 
conversation analysis, some of the things that could be examined include: 

— adjacent pairs - the use of a phrase that elicits a known response (e.g., How you doing? Fine); 

— tag questions - a statement that is turned into a question to elicit a response (e.g., Nice day, isn't it?); and, 

— back channel cues - a signal that the listener is interested and wants the speaker to continue (e.g., smiling 
emoticon). 

The absence or presence of these kinds of indicators would signal if people are working to nurture a conversation or 
discouraging it. 
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Abstract 

The purpose of this study was to describe the 1:1 Digital Classroom Project undertaken with 19 Oklahoma 
schools implementing one-to-one computing with 8 th graders over a 17-month period. Findings from data gathered 
through document analysis are discussed in terms of the key implementation elements of managing change, 
technical infrastructure, professional development, instructional change, and evaluating success. The most important 
aspect of this project was to enhance teaching and learning in each classroom, with a particular focus on 21st 
Century Skills. The 21st Century Skills Assessment and the LoTi Digital Age Survey were used to measure success 
of the project. An Educational Technology Integration Specialist (ETIS) was hired at each school to facilitate 
professional development and support. 



Introduction 

Evidence of the current nationwide call for school improvement can be seen everywhere from morning 
news shows to political town hall meetings to the stress on the faces of teachers and administrators. Educators and 
policymakers are trying to ascertain what is happening or not happening in schools, insure more equitable access to 
computers and information between students from poor and wealthy families, increase student test scores, transform 
the quality of instruction, and prepare students for a future that will undoubtedly feature ubiquitous computing. 

School districts, state initiatives and federal programs have facilitated the implementation of one-to-one 
laptop programs since the turn of the century, and the results are filtering in to inform education theory and practice. 
By the 2004-05 school year, one-third of Maine high schools provided laptops to students. Indiana, Michigan, New 
Hampshire, Texas, and Vermont also began early one-to-one initiatives at the state level, while a myriad of 
individual districts nationwide experimented with local programs (Zucker, 2005). 

While some individual Oklahoma schools had engaged in their own one-to-one computing projects, the 
first statewide initiative began in Spring 2009 with funds from NCLB Title IID ARRA Competitive Grant and under 
the direction of the State Superintendent of Schools. The 1:1 Digital Classroom Project (DCP) awarded a total of 
$6,500,000 to 19 schools for the February 2010-June 2011 one-to-one implementation. Each of the 19 school sites 
selected to receive the grant were expected to target 8th grade students, hire an Education Technology Integration 
Specialist (ETIS), allocate a minimum of 25% of the budget to professional development, measure the 21st Century 
Learning skills of students and teachers, implement the H.E.A.T. (Higher order thinking, Engaged learning, 
Authentic connections, and Technology use) framework for lesson planning, and provide six quarterly and one final 
report. The Oklahoma State Department of Education (OSDE) provided support through a variety of face-to-face 
meetings as well as through an online community (http://onetoonelearning.ning.com). Reviews of research from past 
initiatives reveal that certain elements impact implementation: effective leadership, thorough planning, initial and 
ongoing targeted professional development, buy-in from all stakeholders, a robust infrastructure, and assessment of 
the impact on student learning (Argueta, Huff, Tingen & Corn, 2011). A review and analysis of school-level 
initiatives by the Metiri Group (2010) found that evaluation of one-to-one initiatives should include “multiple 
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assessments to measure academic achievement, 21st century skills, and engagement” (p. 3) in order to provide a full 
picture of student learning. OSDE factored in these lessons learned to their design of the required project elements. 

The purpose of this research study was to describe the implementation of a one-to-one computing project 
initiated in 19 schools throughout Oklahoma. Table 1 offers basic demographic data on the participating schools. 
The average percentage of students on free and reduced lunch for this group is 73.7%, compared to the state average 
of 59% (Education Oversight Board/Office of Accountability, 2010). The school enrollment data shows some 
variance (low of 80 students in School G to a high of 691 in School P), but “school” is interpreted differently. 

School G, for example counts an enrollment of 80 in grades 7-12, while School P’s enrollment reflects grade 8 at 
one of the two middle schools in the district, so there actually is more variance in school size than indicated in the 
table. Fifteen of the 19 districts had fewer than 854 students enrolled, and only two of the districts, School L and 
School P, would be considered a large school within the context of Oklahoma. 





Free & Reduced 
Lunch 


District Population 


School Site 
Enrollment 


School District 
Enrollment 


School M 


77.5% 


1,283 


654 


654 


School A 


83.9% 


9,059 


410 


1,937 


School GV 


77.7% 


4,254 


394 


394 


School B 


93.2% 


2,668 


484 


484 


School L 


56.8% 


107,263 


621 


14,363 


School D 


78.8% 


1,595 


217 


376 


School E 


72.5% 


3,608 


120 


653 


School G 


95.0% 


733 


80 


122 


School P 


67.0% 


29,349 


691 


5,157 


School S 


70.4% 


2,822 


186 


526 


School O 


64.4% 


3,946 


180 


853 


School H 


75.7% 


1,813 


358 


507 


School C 


57.8% 


3,511 


128 


641 


School W 


71.3% 


3,606 


498 


679 


School R 


82.7% 


4,012 


196 


820 


School LL 


54.8% 


11,288 


482 


2,114 


School Y 


87.1% 


415 


62 


85 


School T 


72.0% 


1,383 


182 


287 


School RD 


62.6% 


3,587 


131 


704 



Table 1: Demographic data for participating schools 
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Since the researchers were granted access to the program documentation at the end of implementation, 
document analysis was the appropriate methodology for the study. Each school designated a project director to post 
documents to the 1 : 1 DCP online community, which was organized into forums for each key activity or requirement 
of the grant: “Announcements,” “Site Visits,” “1:1 DCP Reporting,” “RESULTS,” “Pedagogy and Instructional 
Strategies,” and “DCP Bootcamp,” (Hileman, 2011) in addition to a variety of forums for sharing information on 
one-to-one computing research, training opportunities, news, and tips for success. Specifically, the researchers 
analyzed data from grant proposals, quarterly reports, final reports, and assessment reports. Findings from the data 
are discussed in terms of the key implementation elements of managing change, technical infrastructure, 
professional development, instructional change, and evaluating success. 

Managing Change 

One of the first requirements for participating schools was to submit a job description for and hire an 
Educational Technology Integration Specialist (ETIS), reflecting the importance of this technology grant being more 
than just an addition of hardware for schools. OSDE set forth some guidelines for the duties of the ETIS: 

• provide input on professional development topics, schedules, and evaluation; 

• support teachers in technology integration both in-class and out-of-class; 

• identify emerging technologies and digital resources for teachers to use; 

• work with teachers, library media specialists and administration to foster infusion of 21st Century Skills into 

participating classrooms; 

• facilitate meetings with teachers to develop problem-based learning; and 

• attend all 1:1 DCP meetings. 

Fourteen of the 19 schools specified in their ETIS job descriptions that the qualifications included 
certification as a classroom teacher. Six of these specified certification in “elementary or secondary,” while one 
school each were searching for someone with “math or science,” “Library Media Specialist or educational 
technology,” and “career and technical education or early childhood” certifications. While all of the job descriptions 
included the responsibility of providing professional development, three schools specified that the ETIS would offer 
professional development “identified by administration.” Seven schools were very specific that the job would be 
available for “the term of the grant” or even a certain number of days (ex: 240, 180, and 185). While not required by 
the project, 14 of the school wrote into the job description the responsibility for evaluation of grant activities, 
maintaining statistical records, and reporting findings to administration for the purpose of data-driven decision 
making. Schools A, D, and LL added unique duties to their ETIS job description. School A’s ETIS was charged with 
teaching a “7th grade technology class to develop additional technology assistants to support teachers and students” 
and “provide needed skills to master 8th grade national educational technology standards.” The ETIS at School D 
would “assist teachers in adding pages to the school website,” and School LL specified that their ETIS would focus 
on development and implementation of NETS; critical media literacy skills; professional learning communities; and 
inquiry, challenge-based, and problem-based learning. This school clearly had an existing technical support 
structure, as the ETIS was charged with “collaborating with technicians to address pedagogy needs of instructors.” 

Technical Infrastructure 

Each participating school selected the particular hardware for their one-to-one initiative. School G chose to 
purchase iPads, School A selected Neo 2 devices, and Schools RD, Y, W, and L purchased netbooks. The other 13 
schools implemented laptops for their 1:1 DCP project. The RFP for the grant specifically noted the best installation 
would be an 802.1 In wireless network infrastructure, and all of the schools were able to apply for E-Rate funds to 
upgrade their existing network. Five schools cited computers-to-student ratios of 1:18 or more as a significant 
problem for teaching and learning. Two schools cited an outdated infrastructure — School A was stressed to the 
point of one Internet connection per classroom, and students and teachers routinely encountered connectivity 
overloads and shutdowns. 

Two of the biggest challenges that schools faced were how to maintain the computers and manage the 
training the 1 : 1 DCP grant initiated. Hardware breakdowns that stalled instruction and resulted in inconsistent use of 
technology at random times throughout the implementation year was reported as a source of frustration from 
teachers at Schools L, P, S, Y and G. The grant’s requirement of a plan for sustainability encouraged each school to 
explore ways to not only maintain the hardware they had acquired with the grant but to increase the number of 
computers so that more students at more grade levels would have their own devices. Five schools ( C, LL, M, S and 
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W) planned to eventually achieve 1:1 status for every secondary student in the district; School G hoped to achieve a 
computer for each student at every grade level in the district. In their grant proposal. School Y accurately predicted 
that “challenges will lie in making sure all laptops are working and in front of the students. Another challenge will 
be development of policy ensuring that the laptops will be taken home each night to complete projects and then 
returned safely each day.” Four other schools echoed this concern for daily and practical issues students and teachers 
would face in a one-to-one learning environment. While eight schools believed attendance, engagement, tech skills 
and learning would increase through 1:1 DCP, one school was concerned that the addition of computers could also 
cause behavioral problems. 



Professional Development 

Each grant proposal focused on the significance of professional development in accordance with the RFP. 
Professional development activities were required to reflect the integration of technology into the teaching and 
learning process. Each district developed a professional development plan, with common elements emerging across 
participating schools: developing lessons with the Fl.E.A.T. Lesson Plan format, problem-based learning, and 
attendance at a day-long conference that School C hosted. Even considering the short time frame of the grant - 17 
months - the professional development component was expected to occur at an intense rate in order for the grant 
activities to have a significant impact on student learning. The professional development occurred in a variety of 
formats: face-to-face workshops, webinars, and online classes. Each school also administered the LoTi Digital Age 
Survey to participating teachers. This survey offers group recommendations for professional development offerings 
based on the profile resulting from responses. 

In their proposal, School B noted that lack of professional development for technology use was one of the 
most serious obstacles to fully integrating technology into curriculum. The effort of strategic integration required 
more professional development than was currently available at each site. Schools also believed that their training 
needed to be consistent and ongoing rather than just at the beginning of the grant. All schools involved in 1 : 1 DCP 
added an ETIS position and that position was responsible for planning the professional development throughout the 
duration of the grant. 

School Y stated, “school reform must begin with teacher reform. The teachers at the school were well 
aware that simply adding technology would not guarantee [students’ needs were met]. To improve student 
performance, teachers must be trained in delivering technology-based curriculum.” Six of the nineteen schools 
identified professional development as a problem they faced in meeting technology standards. A needs assessment 
conducted by each district indicated that the greatest needs were training on 21st Century Learning skills and Web 
2.0 tools, high quality trainers, and specific technology certifications. School T planned a ‘train the trainer’ model 
for professional development where they sent the ETIS or another teacher to an event or a workshop, and then he or 
she would conduct a workshop locally for colleagues. 

Once the project implementation was under way, schools were quickly aware of the value of meaningful, 
high-quality technology training for teachers. Schools FI, M, C, L and D specified that they wanted more 
information than simply how to use software; they needed information and training about creating curriculum and 
activities that would call for students’ active learning in project-based scenarios and technology integration. School 
S reported achievement of their goal to “build a sustainable model of professional development for teachers 
including peer-mentoring and anytime-anywhere professional interactions (onsite, offsite, and online.)” Per the RFP, 
each site scheduled intense professional development initially to prepare for the infusion of technology, and then 
regular meetings and training sessions were planned on a weekly or monthly basis throughout the duration of the 
grant. 

The ETIS position at each site offered support for teachers and helped teachers to be more prepared going 
into the second year of the 1:1 project. Professional development was created to meet needs at each individual site. 
Teachers at five sites were were concerned about hardware maintenance and the time and technical expertise 
involved in solving daily software and hardware problems. The ETIS from three sites reported that teachers were 
comfortable using their own laptops for developing curriculum and using the digital curriculum; however, three 
schools were experiencing challenges in the paradigm shift from teacher-led to student-led activities. Teachers from 
Schools S, Y and D proposed to find Web 2.0 learning tools and other available software that would support content 
required while integrating technology with curriculum. Timing of professional development during the school year 
was a challenge. In the late spring during preparation for the one-to-one implementation, some teachers and 
principals were focused on preparing students for tests and removed themselves from professional development 
opportunities. Training that was needed for implementing a learning management system was needed in one school, 
but they delayed until the first summer following the 1:1 implementation. The schools’ final reports indicated an 
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overall positive response to technology integration with some degree of hesitation due to the time involved, the 
collaboration commitment, and the changes from a traditional learning environment to a technology-rich learning 
environment. 



Instructional Change 

Theories that informed the grant application and implementation process were intended to transform 
practice from a teacher-centered paradigm to a student-centered paradigm by developing a technology-rich 
environment to improve student achievement in multiple areas. Throughout each schools’ proposal, the terms 
immersion, integration , and engagement were used to describe the plans schools had in place to change instruction 
from one paradigm to another. Authentic learning activities adopted by individual classrooms as well as entire 
schools lead to greater student understanding and higher levels of creativity (DiMartino & Casteneda, 2007). Online 
research, mentoring, and peer review activities were also encouraged through professional development before the 
implementation and throughout the school year. School P gave students 24/7 access to their computers specifically 
to encourage their continued learning beyond the school day, thus increasing the possibility of additional learning 
(Marzano, 2000). To implement collaborative instruction focusing on 21st Century Skills in a one-to-one computing 
environment, teachers had to make changes in their approach to education away from the more traditional didactic 
model to a newer model that would use digital literacy to promote anytime, anywhere learning. 

The schools’ proposals cited problems in the areas of student engagement, teacher motivation and teacher 
preparation as concerns for instructional change. Eight of the schools identified achievement as a critical problem 
for their schools and needed to meet proficiency status on Oklahoma Core Curriculum Tests. Educational problems 
of equity of access proved to be the concern of seven school districts with high percentages of free and reduced 
lunch counts and Title One status. One of these five districts indicated that lack of technological equity was a 
problem due to the poverty of the students (School B: 21% poverty rate and 93.2% Free & Reduced Lunch) and 
remote geographical location. Rural communities are declining in numbers and in socioeconomic status, and their 
student population is feared to be at risk for academic failure as well as failure to succeed in the workforce after 
public school. Schools C and D reported being unable to maximize homework assignments because these students 
had extremely limited access to computers outside of school as well as inside of school. 

Schools with a better computer-to-student ratio felt the need for a more comprehensive program where 
teachers and students not only had the computers 24/7 but had the support necessary for teaching and learning on the 
computers as well. School L had a lower computer-to-student ratio, but their proposal indicated concern that 
students were not accustomed to using computers for academic purposes. Three schools’ students were maintaining 
a steady level of achievement - no growth and no decline - but these schools had set long-term goals of technology 
literacy by the end of their eighth grade year. Also, these same schools believed that their teachers did not have the 
training they needed to embed technology in their classroom curriculum. School GV felt like their teachers needed 
to be “challenged to reevaluate, rewrite, recreate, and revolutionize the manner in which [they] taught [their] 
students.” 

Two schools suggested that they were economically disadvantaged and believed that they were at risk for 
success due to the high student to computer ratio. Five schools, including School S, stated clearly that putting a 
computer in the hands of every student, 24/7 would empower students to take a more active role in their education. 
School W had experienced success with teachers’ willingness to “use laptops for instruction... check out mobile labs 
of laptops for student use. ..engage students in Smart Board interaction, and utilize computers that were installed in 
their classroom” and they believed that their teachers would embrace a 1: 1 initiative because teaching and learning 
would be more productive since students had access to computers constantly at school and at home. Disengaged or 
failing students were represented by high absenteeism rates and discipline problems. In School M, students averaged 
21.5 written discipline reports per student grades 5-8 and an alarming rate of high school dropouts. Lack of 21st 
Century technology skills was also seen as a huge problem for students as well as teachers. 

The schools also spoke to the issue of true integration of technology and learning (Schools H, M, C, D and 
L). The hope was that with training, true integration of technology and learning would occur by making these 
devices and individual computers part of student-centered learning where interaction between students and media 
were commonplace. Schools viewed the 1:1 program as a potentially significant step in helping student become life- 
long learners as they learned how to use computers for activities beyond social networks: their desire was to build a 
community of leaders and learners that collaborate with others to solve problems, make decisions, and create 
knowledge through use of technology. 

All schools had some degree of access to technology for lessons prior to receiving the grant, but their 
students had varying degrees of access to computers, which limited the flexibility of classes to move at a productive 
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pace while sharing or waiting on computers. School G’s teachers who were not comfortable with computer 
technology and wanted to maintain traditional teaching methods to ensure that students were prepared for the 
achievement tests at the end of the year were seen as an impetus for instructional change. Two schools had minimal 
bandwidth available, which caused connection gluts and lost connections for teachers. Teachers would lose 
information or lesson presentations would stall when the Internet was overloaded, and the system would shut down. 
Technical difficulties due to lack of connectivity and/or support caused much frustration for Schools A and B. 

Once the grants were awarded, the first task for each school was to select a curricular focus. Reading and 
critical thinking skills were the most selected to concentrate on for the duration of the project. All topics chosen and 
the schools associated included: 

Reading - Schools A, B, C, G, O, W, and Y; 

Critical thinking skills - Schools LL, RD, D, GV, H, and W; 

21st Century Learning skills - Schools A, LL, and R; 

Math - Schools A, O, and Y ; 

Technology - Schools O, T, and Y; 

Writing - Schools E and G; 

Building academic vocabulary - Schools M and P; 

Digital literacy - School C; 

Communication - School H; and 

Higher order thinking skills - School S. 

Selecting a curricular focus at the beginning of the grant gave each school a framework to build their H.E.A.T. 
lessons and professional development around. 

By way of the ETIS, schools challenged their teachers to use wireless computing devices to instinct 
students 50% or more of their instructional time as documented in lesson plans and in meeting NETS-T standards. 
Schools also anticipated positive results from the curriculum testing students would participate in the last quarter of 
the academic year. 

The challenge teachers faced as they shifted from teacher-led to student-engaged classrooms was the 
central issue before the grant was initiated, and, based on the results following the first four or five quarters of the 
1 : 1 DCP, this challenge remained throughout the life of the grant. The comfort level in a high-tech classroom 
environment for teachers varied: some were delighted with the challenge while others struggled. Teachers 
encountered change in the status quo, which lead to frustrations with technology and fear of the implications the 
change in pedagogy would have on the testing results. The challenges of student maintenance and management of 
computers were considered before the grants were implemented, and all participants - teachers, parents and students 
- attended meetings and signed agreements to prepare for the advent of the individual computers into their schools. 

Teachers anticipated that if each student had a computer to use throughout the school day as well as at 
home in the evening and on weekends, grades would increase, a love of learning would evolve, and that students 
would improve in their technology skills as well as in their level of self-confidence. Other teachers were concerned 
about the change in classroom management a one-to-one computing environment would bring about. Teachers and 
administrators both realized the need for collaboration but also the challenges that collaboration would add to an 
already-busy daily schedule. School L summarized the concerns expressed by four other schools: “Teachers [will 
need to] find their own comfort level with the 1 : 1 digital classroom environment and the integration of new 
technologies.” 



Evaluating Success 

Overall, schools anticipated that a one-to-one implementation would yield higher achievement and greater 
comprehension and learning for all students. According to their proposals and goals, they believed that students 
would grow in technical literacy and use the instant communication features that having a computer 24/7 would 
allow. They planned to initiate such a project with various types of professional development programs. 

The primary focus of the long term goals for nine schools was increased academic achievement through 
software, online resources, and appropriate learning technologies that have been shown to improve student academic 
achievement and technology literacy through a sustained one-to-one initiative. Schools are under pressure to 
increase achievement — particularly in math and reading — and to make Adequate Yearly Progress (AYP) per No 
Child Left Behind (NCLB) and State requirements. Schools projected teacher and student successes in terms of 
engagement, reduced absenteeism and reduced tardiness, increased hands-on learning time, opportunities for online 
collaboration, synchronous and asynchronous feedback, and increased technology skills. 
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The 21st Century Learning Skills Assessment was one of two instruments used to measure the success of 
this project. It is designed to provide deeper insight into students' and teachers’ grasp of critical 21st century skills: 
creativity, innovation, information fluency, critical thinking, decision making and digital citizenship and how they 
compare to national averages. Proficiency is plotted across four levels: below basic, basic, proficient, and advanced. 
Table 2 offers overall proficiency scores for students and teachers in each school as well as the category in which 
students and teachers scored lowest and highest. Across all schools, Creativity & Innovation and Communication & 
Collaboration were the lowest-scoring categories. Students scored highest on Digital Citizenship in 11 of the 19 
schools, and teachers scored highest in the category of Critical Thinking, Problem Solving & Decision Making. 
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Table 2. 21st Century Learning Skills Assessment proficiency scores and lowest/highest categories by students and 
teachers per school 



Perhaps the most interesting aspect of these assessment results is that the teachers’ level of proficiency was 
at or above the students’ level of proficiency at each school. Authors Marc Prensky (2001) and Don Tapscott (1998, 
2008) have generated widespread discussion of “digital natives” and their expertise with technology compared to 
“digital immigrants.” More recently, researchers have begun to challenge the idea that the younger generation is 
more adept with technology (Haigh, 2011; Bennet, Maton, & Kervin, 2008; Vaidhyanathan, 2008), and that 
viewpoint is gaining momentum. The data encountered in this study on the 21st Century Learning Skills Assessment 
clearly reveals a higher level of proficiency in teachers. 

Each participating school was also required to administer the LoTi Digital Age Survey (Moersch, 1995) to 
participating teachers. The purpose of this three-part instrument is to measure classroom teachers’ implementation of 
the tenets of digital-age literacy as manifested in the the five domains of the National Educational Technology 
Standards for Teachers (NETS-T): Digital-Age Work and Learning, Digital Age Learning Experiences and 
Assessments, Student Learning and Creativity, Professional Growth and Leadership, and Digital Citizenship. The 
three sub-surveys are the Levels of Technology Innovation (LoTi), Personal Computer Use (PCU), and Current 
Instructional Practice (CIP). The LoTi measures “the transformation from didactic teaching practices and student 
compliant learning to digital-age teaching and learning characterized by the use of digital tools and resources to 
promote higher order cognitive processing, engaged student learning, and authentic, real-world problem-solving” 
(Moersch, 1995). PCU determines the intensity level of the teachers’ fluency with using digital tool and resources 
for learning, and CIP plots teachers along a continuum from low intensity (no instruction occurring at all) to high 
intensity (a highly flexible student-centered instructional approach characterized by diversified and authentic 
activities driven by student questions). 

Each school received results for the LoTi, PCU, and CIP individually as well as an overall Digital Age 
Profile indicating their need for improvement or priority focus for professional development. Low signifies little 
need for professional development, while High indicates a great need in that category for that school. Table 3 details 
results for each of the sub-instruments, the mean for those results, and the areas suggested for each school to focus 
on as professional development priorities based on the results of the sub-instruments. Data were collected at the end 
of the implementation school year. 
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Table 3. LoTi Digital Age Survey results by school 
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Only two schools received a Low on any area in their Digital Age Profile, indicating that they did not need to focus 
on professional development in that category. School G scored Low on Digital Age Learning Experiences & 
Assessments, and School H received a Low rating on Professional Growth & Leadership. The Highest need area for 
professional development across all schools was in Student Learning & Creativity, and the lowest need was in the 
area of Professional Growth & Leadership. 

For the LoTi portion of the assessment, 17 out of the 19 schools scored at LoTi Level 2: Exploration. At 
this level, 

the instructional focus emphasizes content understanding and supports mastery learning and direct 
instruction. Teacher questioning and/or student learning focuses on lower levels of student cognitive 
processing (e.g., knowledge, comprehension). Digital tools and resources are used by students for extension 
activities, enrichment exercises, or information gathering assignments that generally reinforce lower 
cognitive skill development relating to the content under investigation. There is a pervasive use of student 
multimedia products, allowing students to present their content understanding in a digital format that may 
or may not reach beyond the classroom. (LoTi, Inc., 2011) 

The mean PCU score across schools was a 2.26 with a mode of 2. PCU Intensity Level 2 (Not True of Me 

Now) 

indicates that the participant demonstrates little to moderate fluency with using digital tools and resources 
for student learning. Participants at Intensity Level 2 may occasionally browse the Internet, use email, or 
use a word processor program; yet, may not have the confidence or feel comfortable using existing and 
emerging digital tools beyond classroom management tasks (e.g., grade book, attendance program). 
Participants at this level are somewhat aware of copyright issues and maintain a cursory understanding of 
the impact of existing and emerging digital tools and resources on student learning. (LoTi, Inc., 2011) 

The CIP scores were slightly higher, with a mean of 3.16 and a mode of 3. This indicates 
an Intensity Level 3 (Somehat True of Me Now), where 

the participant supports instructional practices aligned somewhat with a subject-matter based approach to 
teaching and learning — an approach characterized by sequential and uniform learning activities for all 
students, teacher-directed presentations, and/or the use of traditional evaluation techniques. However, the 
participant may also support the use of student-directed projects that provide opportunities for students to 
determine the “look and feel” of a final product based on their modality strengths, learning styles, or 
interests. Evaluation techniques continue to focus on traditional measures with the resulting data serving as 
the basis for curriculum decision-making. The use of research-based best practices expands beyond basic 
classroom routines (e.g., providing opportunities for non-linguistic representation, offering advanced 
organizers). 

Overall, results indicate very low scores on each of the 0-7 scales. Means from Schools G (4), C (3.67), and 
B (3) were on the higher end among the 19 participating schools, but still quite low considering the assessments’ 
scales. The lowest means were from Schools P (1.33) and W (1.67). 

Conclusions 

In their applications for the 1:1 DCP grants, schools identified students’ lack of access to computers on a 
consistent, continual basis as their primary concern for readiness to meet the demands of the 21st century upon 
graduation from high school. Schools also identified the importance of raising test scores and improving student 
achievement. Parallel to the learning issues was the expressed commitment needed for “extensive” (School A) 
professional development. The intention for integration was stated repeatedly as the key component in facilitating an 
engaging curriculum that produced better test scores. While there were different expectations about the amount of 
increase, each of the 19 schools in the 1:1 DCP grant believed that success with this infusion of technology into their 
programs would yield increased academic learning, and higher test scores. (Note: As standardized test scores 
become available for the implementation year, the researchers will examine gain scores in comparison to the year 
previous to implementation.) 

The purpose of this study was to describe the 1:1 Digital Classroom Project undertaken with 19 Oklahoma 
schools. The schools selected represented diversity in size, population, and current technology use. Four of the 
districts had received past recognition for one or more of the following awards: “Technology District of the Year,” 
“Technology Leader of the Year,” and “Technology Teacher of the Year.” In setting forth guidelines for 
participating schools, the OSDE followed lessons learned from other states in the areas of managing change, 
technical infrastructure, professional development, instructional change, and evaluating success. The ETIS at each 
school was hired to lead professional development, support teachers in and out of the classroom, and attend all 
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OSDE meetings. ETIS personnel also participated in the online community developed for project tracking, 
document sharing, and communication. Another required factor designed to help manage change was involving non- 
school entities in development of policies, procedures, and handbooks at each school. 

Each of the schools had some existing technical infrastructure available to build upon. Most schools 
purchased laptops (a mix of Mac and Windows OS), but others chose netbooks, Neo 2 devices, or iPads. Upgrading 
wireless routers and access points were common expenditures in each of the schools. Professional development 
began with hardware and software training, and the biggest challenge in early stages of the grant was solving 
Internet access issues. Each school designed their own professional development plan, and the OSDE facilitated 
opportunities for all of the schools to engage in particular events such as training to use the H.E. A.T. lesson plan 
framework (LoTi, inc., 2011). 

The most important aspect of this project was to enhance teaching and learning in each classroom, with a 
particular focus on 21st Century Skills. Additionally, each school selected a curricular focus, around which they 
would design collaborative, cross-curricular lessons. The 21st Century Skills Assessment and the LoTi Digital Age 
Survey were used to measure success of the project. The results of the 21st Century Skills Assessment revealed that 
students tested at Basic proficiency, while teachers scored at the Proficient level. None of the schools’ teacher or 
student groups were considered to be Advanced when assessed at the end of the project. Creativity & Innovation and 
Communication & Collaboration were the two lowest-scoring categories. Students scored highest on Digital 
Citizenship in 1 1 of the 19 schools, and teachers scored highest in the category of Critical Thinking, Problem 
Solving & Decision Making. Since critical thinking and problem based learning were key elements of 1:1 DCP 
activities, it is not surprising that teachers scored highest in this area. However, additional examination should be 
conducted to determine why the transference to students’ knowledge and skill was not apparent. Another factor 
emerging from this assessment data was that teachers scored higher than students. This also merits further 
investigation to perhaps dispel the “digital native/digital immigrant” myth and find practical ways to increase 
teachers’ confidence in their own technology use. 

Digital Age Profiles were determined from participating teachers’ scores on the LoTi Digital Age Survey. 
Profiles revealed that professional development should particularly focus on the weakest area of Student Learning & 
Creativity. Scores were particularly low on the LoTi and PCU instruments but marginally higher on the CIP 
assessment. Ideally, this assessment would be completed at the beginning of the project and again at the end for 
comparison. Since data presented was collected approximately one year after the project began and after teachers 
had made a significant commitment to professional development and instructional improvement through the use of 
technology, the researchers expected much higher scores. However, the data does give schools a road map to 
continue the growth started through this grant project. 

Educators continually face the challenge of teaching students effectively and reaping tangible results of 
their learning. Oklahoma’s 1:1 DCP grant enabled 19 schools to decrease the digital divide, embrace instructional 
change, experience problem-based, cross-curricular instruction, and prepare students for anytime, anywhere 
learning. Sharing the implementation of this project contributes to the growing body of literature on one-to-one 
computing initiatives and generates additional questions for further research. The project itself may have served as a 
solid foundation upon which teachers and students in these schools can continue to build knowledge and skills for 
digital age work and learning, creativity, professional growth, leadership, and digital citizenship and responsibility. 
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Abstract 

This paper compares the perceptions of two groups of students who participated in the first cohort of the WebIT 
online Masters of Science Degree (M.S.) in IT program, at The University of Tennessee at Knoxville during 
2008-2010. The first group, the program COMPLETERS (N=l 1), are the students who completed the WebIT 
program and graduated at the end of Spring semester, 2010. The second group, the DROPPER-respondents (N=5), 
are those students who dropped out of the WebIT program and elected to complete a survey about their WebIT 
experiences. These comparisons illustrate several possible differences between the two groups that provide insight 
into the high rate of attrition observed during the first cohort of the WebIT program. 

Keywords: online program attrition, online student retention 

A Comparison of Successful and Unsuccessful Online Students: 

Implications for Reducing Student Attrition 

Twenty-five students were recruited during the Spring of 2008 for the first cohort (WebIT 1) of the WebIT 
online M.S. program in Instructional Technology at The University of Tennessee at Knoxville. Four students 
dropped out before the academic portion of the program began, 1 student dropped out during the first semester, 3 
students dropped out during the second semester, 4 students dropped out, and 1 was lost due to academic difficulties, 
during the third semester, and 1 additional student was lost due to academic difficulties during the fifth semester. Of 
the original 25 students who began the program, 1 1 graduated at the end of Spring semester, 2010. The overall 
attrition rate during WebIT 1 was 56%. Attrition rates reported for other online programs vary widely, but similar 
rates are relatively common (Bowser, 1992; Carr, 2000; Chyung, Winiecki & Fenner, 1998; Diaz, 2002; Kember, 
1989; Moore, Bartkovich, Fetzner & Ison, 2002; O’Brien, 2002; Rovai & Downey, 2005; Rovai & Wighting, 2010). 

The WebIT 1 cohort group completed several anonymous, electronic surveys to provide feedback to the 
WebIT curriculum designers, instructors and IT faculty. The data from these surveys provide information regarding 
student perceptions of their expectations and actual online program experiences and offer clues as to why so many 
students failed to complete the WebIT program. 

The purpose of this paper is to share several interpretations of the data provided by the WebIT 1 students 
(both those who dropped and those who completed the program). These interpretations are based on a small sample 
of students and thus any generalization from these findings should be made with caution. While the issues identified 
may be of more widespread applicability, they also may be somewhat idiosyncratic to the specific sample of 
individuals who participated in the WebIT 1 cohort. These interpretations are being shared in hopes that they may be 
of use to others currently involved in the delivery of online programs or those who may be interested in developing 
new online programs. 
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Method 



Study Participants & Curriculum Context 

Twenty-five students were recruited to the WebITl cohort of the WebIT program in the spring of 2008. 
Twenty-one students matriculated beginning in the summer of 2008. Of these 21 students, 16 were teachers and the 
remaining 5 worked in other occupations. During the first year of the program, 8 students dropped out and 1 student 
was removed for academic difficulties. During the second year of the program 1 additional student was lost due to 
academic difficulties. Of the 10 WebITl students who began but did not complete the program, 6 were teachers and 
4 worked in other occupations. Of the 1 1 WebITl students who completed the program, 10 were teachers and 1 
worked in a technical support field in higher education. 

The WebIT curriculum consisted of 1 1 semester-length (3 credit) courses. The first course was delivered in 
Summer semester, 2008. After that, two courses were taught during each semester for five consecutive semesters 
(Fa08, Sp09, Su09, Fa09, SplO). Waugh, DeMaria and Trovinger (2010) describe the WebIT curriculum in greater 
detail. 

Data Sources 

A total of 14 students were lost from the WebITl cohort of the program, 13 during the first year and one 
additional student during the Fall semester of the second year. As mentioned previously, 4 students dropped out 
before the academic portion of the program began; they applied and were admitted to the program, but never 
matriculated. This group of 14 students will be referred to as the DROPPERS. Five of the 10 DROPPERS who 
matriculated in the WebIT program elected to complete an anonymous, electronically-administered program 
completion survey. This group of 5 students is referred to as the DROPPER-respondents group. Eleven students 
completed the WebIT curriculum and elected to complete an anonymous, electronically-administered program exit 
survey. This group of 1 1 students is referred to as the COMPLETERS group. 

Assessments and Measures 

A team of faculty-researchers working with the WebITl cohort developed the two surveys that were used 
to gather the student data discussed in this paper. The WebIT Online Program Attrition Survey (PAS), was 
administered during Fall 2009 to the DROPPER group. The WebIT Online Program Completion Survey (PCS), was 
patterned after the PAS so as to contain identical items for comparison. The PCS was administered to the 
COMPLETERS at the end of Spring semester, 2010. Each survey contained 46 comparable items. Most of the 
comparable items were identical in wording but some were minimally altered to ensure proper grammatical format. 
This paper will discuss the data provided in response to 10 items that were common to both surveys. 

The two surveys were designed to address specific items of interest to the research team, and thus were 
judged to possess face validity in this regard. With regard to reliability, each survey instrument contained several 
items of a similar nature and these items were used to assess consistency in student response patterns, thus enabling 
a simple estimate of reliability, or internal consistency. Though these estimates of validity and reliability are not 
robust, they do serve as some assurance that the characteristics of the instruments were sufficient to address the 
purposes of this project. 

Research Design 

The research design employed for this study is a case study model (Stake, 1995). The following questions 
guided the analysis. With regard to their experiences and participation in the WebIT program, how do the 
perceptions of those students who complete the WebIT program compare with those who fail to complete the 
program? Does this comparison offer any insight regarding possible ways to reduce future attrition? 

Results and Discussion 

The WebIT student feedback provides WebIT planners with information to guide ongoing program 
development efforts, but it also may provide insight applicable to other online program initiatives. While the dataset 
being analyzed is limited, this fact in no way negates the potential for broader applicability of the findings. 
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Student Expectations of the WebIT Experience 

Reasons for Choosing WebIT. Both the DROPPER-respondents and the COMPLETERS were asked to 
rank order a list of possible reasons for why they chose the WebIT program. The choices were: (a) academic 
program is offered in online/electronic format, (b) value of the academic program in meeting employment goals, (c) 
flexibility of academic program in fitting my schedule, (d) time duration of academic program: 2 years, fixed pace, 
(e) academic rigor of program, (f) cost to complete the academic program, and (g) availability of financial aid 
support. A simple majority of students in both groups (6/1 1 COMPLETERS, 3/5 DROPPER-respondents) awarded 
their top rank to choice (a), the online format of the program, but other choices were also top-ranked. For the 
COMPLETERS, the other two top-ranked choices were (b) and (c), each receiving 3 and 2 responses, respectively. 
For the DROPPER-respondent group, the other choice that was top-ranked was (c), receiving 2 responses. The 
online format of the WebIT program was the top reason why students elected to matriculate in the program. Based 
on the minority responses, some students saw the WebIT program as being flexible/better fitting their schedules 
(COMPLETERS ranked third overall, DROPPER-respondent group ranked second overall); and some students saw 
the WebIT program helping them meet employment goals (COMPLETERS group ranked second overall, 
DROPPER-respondent group ranked fourth overall). Respondents seem to have chosen the WebIT program because 
it was offered in a format that was appealing to them given that all (16/16) work full-time (15 were teachers). From 
the outset, they saw the online format of the WebIT program as being more flexible and permitting them to meet 
employment-related goals. 

Life Priorities. Both groups were asked to rank items in a list to identify which were priorities for them as 
they participated in the WebIT program. Both groups identified the same three factors, in the same order, as their 
top-ranked priorities, (a) family, (b) work, (c) WebIT. The COMPLETERS assigned top-ranks as follows: (a) 
family, 9/11; (b) work, 2/11. The DROPPER-respondent group assigned top-ranks as follows: (a) family, 4/5; and 
other (Church), 1/5. None of these students identified the WebIT program as his/her top priority, however in terms 
of average rankings, the WebIT program was identified as the third most important priority for both groups. 

Clearly, all of the survey respondents ranked family above all other life-priorities, with graduate study in 
third place. It appears that students saw the WebIT program as a way to pursue a graduate degree by utilizing 
whatever free time they could allocate to the effort while continuing to work full-time and remaining fully engaged 
in family responsibilities and their normal, everyday lives. 

Perception of Workload. Online programs are typically advertised as being more convenient for students 
to complete. One consequence of this might be that students might confuse this idea of convenience in accessing 
graduate study with the degree of difficulty associated with completing the required work. 

To examine this issue, both groups of students were asked to share how their initial expectations of the 
work required in WebIT compared to their actual experiences during WebIT. All 5 members of the 
DROPPER-respondent group and 7 of the 1 1 COMPLETERS (75% of total respondents) responded that the work 
required to successfully complete the WebIT curriculum far exceeded their initial expectation of the amount of work 
that would be required in the WebIT program. This may have been the result of poor communication between 
WebIT planners and potential students during recruitment, or it may be a deep misunderstanding about the difficulty 
of graduate education, or it may be that regardless of how such information is conveyed to potential students, their 
motivation to complete a graduate program through a mechanism that makes it possible (theoretically) to complete a 
graduate degree without disengaging from their current life obligations and relocating to a physical campus is so 
strong, that they are willing to seek out such an option regardless of their perception of the likely difficulty of the 
task. 

Based on our sample, a large proportion of students desiring to complete an online program of study might 
underestimate the likely amount of work that is required to complete an online graduate program. If this is true, then 
online program planners should make every possible effort to communicate a clear picture of the amount of effort 
that will likely be required of students to successfully complete the online program. Success in this communication 
effort might help reduce student attrition. 

Perception of Time Required to Participate. A related question asked respondents about how their initial 
expectation of the amount of time they would need to spend to successfully complete WebIT work compared to their 
actual experiences in WebIT. Again, all 5 of the DROPPER-respondent group indicated that the time they were 
required to spend engaged in academic work during WebIT exceeded their initial expectations. Ten COMPLETERS 
reported that the time required exceeded their initial expectations and 1 of the COMPLETERS reported that the time 
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required was less than s/he initially expected. In response to this question, 94% of the survey responders 
underestimated the likely amount of time required to successfully complete the WebIT curriculum. The results from 
this question reinforce those from the previous question as amount of work required and the amount of time required 
to complete work are related factors. Again, this finding underscores the need to clearly communicate to potential 
online students about how participation in an online graduate program of study will likely impact their lives. If 
students understand that in order to be successful in an online graduate program, they will be required to spend a 
significant amount of time and effort engaged in academic pursuits, this may enable them to make a more informed 
decision regarding how such a course of action might affect their life priorities. 

Time Allocated to WebIT Work. How much time per week did the WebITl students allocate to 
completing WebIT work? Most COMPLETERS (7/11) reported spending more than 15 hours per week on WebIT 
work. Two COMPLETERS reported spending in excess of 25 hours per week. The remaining 4 COMPLETERS 
reported spending between 11-15 hours per week. The majority of the DROPPER-respondents (3/5) reported 
spending between 11-15 hours per week, while the remaining 2 DROPPER-respondents reported spending less than 
10 hours per week. Overall, the COMPLETERS reported spending more time to complete the WebIT work than the 
DROPPER-respondents. Over half of the COMPLETERS reported spending more than 10 hours more per week on 
WebIT work, than the DROPPER-respondents. 

As mentioned above, 75% of the survey respondents reported that the actual amount of work required to 
successfully complete the requirements of the WebIT online curriculum exceeded their initial expectations of what 
would be required of them. Estimates of how much actual work was required range from less than 10 hours per 
week (2 DROPPER-respondents) to more than 25 hours per week (7 COMPLETERS). Both groups also reported 
that WebIT was their third most important priority behind family and work. Because of this, not surprisingly, the 
survey respondents were nearly unanimous (81%) in their view that the most significant challenge they faced during 
their participation in the WebIT program was personal time management (COMPLETERS, 9/11; 
DROPPER-respondents, 4/5). Students were committed to family and work responsibilities, but in order to find 
enough time to complete the WebIT work, some time had to be taken from those two critical responsibilities. 

Several student quotes will illustrate their struggle with this phenomenon. 

DROPPER-respondent 1: While enrolled in this course, I feel like my job and my students suffered... much 
more than I was prepared to accept. It required a lot more time than I had expected, quite possibly because I 
was not as prepared as I thought for the challenging classes. 

DROPPER-respondent 2: 1 feel that the work load [sic] was at times heavy for full time working 
professionals. I am very devoted to my job and feel that I could not do my best work in the program and at 
my job. They were both suffering. (Not to mention my family. )[s/c] In the end, I had to make a decision. 

My plate was just too full. 

COMPLETER 1 : My weekends were spent mostly on course assignments, did not go out much or spend 
enough time with family. 

COMPLETER 2: Personal time to relax and recharge became a priority. I would need to set a side [sic] one 
evening a week to not work on school otherwise it would have consumed ALL of my free time in the 
evenings. 

Student Preferences for Attributes of the WebIT Program 

The responses of the WebIT students to the design of the WebIT curriculum provides data to illustrate that 
students have different preferences for some online program characteristics. These differences may help to partially 
explain the high attrition rate experienced during WebITl. However, high attrition is not simply explained by 
noting varied student preferences. Most likely, the attrition rate experienced by WebIT was influenced by a 
combination of factors such as the following: (a) the degree to which student career goal matches with program 
content, (b) student financial situation, (c) student preferences for program organizational characteristics, (d) student 
skills and experience in IT, (e) student facility with electronic communication tools, (f) student learning 
style/preferences, (g) student motivation, (h) student ability to manage personal time, (i) student family obligations, 
and (j) student work obligations. This is not to say that the program structure and design of the WebIT curriculum is 
irrelevant to the issue of student engagement and retention. Rather, while the program structure and design of the 
WebIT online program are important for student success, it is possible that they may not be the most critical factors 
in determining student retention. 
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The paragraphs that follow report and discuss student responses to several survey questions regarding 
aspects of the organizational structure of the WebIT program. The two groups of survey respondents differ in many 
of their responses to these questions. These differences may provide insights regarding how to recruit and retain 
more potential online program applicants. 

Preferred Degree of Instructor Control over Work Pattern. Both groups were asked to rank a set of 
choices that reflect their preferences for work pattern (individual effort versus group effort) combined with the 
degree of instructor control. Specifically, they were asked to rank order several alternatives in terms of how they 
valued each instructional strategy. Each choice combined two aspects from the following categories: (a) working 
independently, working as members of a small workgroup, working as members of a class cohort; and (b) 
completing work on a schedule determined by the instructor, or negotiated with the instructor. The specific question 
to which the members of the group responded was the following: “Online programs often utilize a mix of teaching 
strategies to help students succeed in learning the content material. Please rank the following teaching strategies 
used in WebIT to indicate how you valued them”. 

Table 1 summarizes the differences between the COMPLETERS and DROPPER-respondent groups in 
terms of the single choice that each respondent identified as his/her top-ranked choice, i.e., the instructional strategy 
that each individual valued above all others. Looking at this table, one can see that the two groups of students tended 
to value the instructional strategies somewhat differently. Nearly half of the COMPLETERS valued most highly that 
the program allowed them to work independently (5/11) to complete work on a schedule determined by the 
Instructor (7/11) (their rankings cluster in the top half of the table); whereas nearly half of the 
DROPPER-respondents valued most highly that the program allowed them to work independently (2/5) and in a 
manner that would allow them to complete work on a schedule negotiated with the Instructor (3/5) (their rankings 
spread more but tend to cluster in the bottom half of the table). This means that WebIT was at least partially 
successful in being flexible enough to address the needs of all of the students, but it appears that some of the 
students want/need a program that is very flexible whereas others are far happier with a greater degree of 
externally-imposed structure. In the context of the WebITl group, a much larger proportion of the COMPLETERS 
expressed a preference for a more structured program organization, whereas a slightly larger proportion of the 
DROPPER-respondents expressed a preference for more flexibility. It is possible that this specific student 
characteristic — preference/need for program flexibility — is important in terms of attrition or retention but such an 
interpretation is merely suggested by these data. It is likely that this is an important variable in determining student 
retention but such a determination will have to await the results of further studies. 
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Table 1 



Comparison Between COMPLETERS and DROPPER-respondents 
on Preferences for Degree of Instructor Control over Work Pattern 



Survey 
Question 
Response Choice 


C 

Top 

Ranks 


D 

Top 

Ranks 


Students work independently 
and complete work on a schedule 
determined by the instructor. 


5 


0 


Students work as members of a small workgroup 
and complete work on a schedule 
determined by the instructor. 


1 


1 


Students work as members of a class cohort 
and complete work on a schedule 
determined by the instructor. 


1 


0 


Instructors specify the exact tasks 
for students to complete. 


1 


1 


Students determine specific tasks 
to meet instructor criteria. 


1 


0 


Students work independently 
and complete work on a schedule 
negotiated with the instructor. 


1 


2 


Students work as members of a small workgroup 
and complete work on a schedule 
negotiated with the instructor. 


1 


0 


Students work as members of a class cohort 
and complete work on a schedule 
negotiated with the instructor. 


0 


1 


Total 


11 


5 



Note. C = COMPLETERS; D = DROPPER-respondents 
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Preferred Work Pattern. In a related question that isolated the concept of preferred work pattern, both 
groups of students were asked to identify their preference in terms of work pattern. In response to the question, “As 
a student in an online program of study, I would prefer to work: (a) independently on some instructional projects and 
collaboratively on some instructional projects, (b) independently when completing instructional projects and tasks, 
or (c) as a member of a collaborative group when completing instructional projects and tasks”, the two groups 
responded somewhat differently. The COMPLETERS group strongly favored a mixed model and selected the 
choices as follows: choice (a), 8/11; choice (b), 2/11; and choice (c), 1/11. The DROPPER-respondents selected the 
choices as follows: choice (b), 3/5; choice (c), 1/5; and choice (a), 1/5. The COMPLETERS seemed to favor a mixed 
approach in terms of work pattern with a preference for some individual and some collaborative activities; whereas a 
simple majority of the DROPPER-respondents seem to favor working individually over engaging in collaborative 
activities. One possible interpretation for this apparent difference is that engaging in collaborative work is more 
difficult and time consuming than doing work alone and those online students who are unable to allocate the 
necessary time to fully engage in collaborative learning efforts do not appreciate the value that such a learning 
strategy can offer. 

Program Organization. In response to the question, “As a student in an online program of study, I would 
prefer a program organization structure that would: (a) provide a logical sequence and pace for me to complete the 
program in a specified period of time, or (b) allow me to complete the program at my own pace, the two groups 
again responded somewhat differently. The COMPLETERS were unanimous (11/11) in selecting choice (a); 
whereas the DROPPER-respondents were split with a simple majority (3/5) preferring choice (b). Again, the likely 
interpretation for this response by the DROPPER-respondents is that they would prefer more flexibility in 
determining their pace of work in completing the online instructional program. The COMPLETERS appear to 
prefer a program that provides a specific work pace that leads to completion of the program in a set period of time. 

Preferred Pacing. Probing the issue of pacing a bit more, both groups were asked specifically about the 
semester course load they would prefer. Specifically, the students were asked, “If the WebIT program could offer 
you the choice, how many online classes would you prefer to complete each semester?” The choices offered were: 1, 
2, 3, or 4 courses per semester; or “variable, depending upon job and personal responsibilities”. The COMPLETERS 
were nearly unanimous (9/11) in selecting 2 courses per semester [the pace required in WebIT]; the other two 
COMPLETERS elected “variable. . .”. The DROPPER-respondents were more mixed in their responses with 1 
wanting to complete a single course per semester, 2 wishing to complete two courses per semester, and 2 electing 
“variable. . .”. The diversity of responses from the DROPPER-respondent group reinforces the interpretation that 
this group would prefer an online program structure that is more flexible in permitting them to complete work when 
and as they can rather than requiring them to adhere to a more focused structure that might require a pace or type of 
engagement that would prove difficult for them. 

Personal Learning Style. Each survey respondent was asked whether or not the overall structure of the 
WebIT program suited his/her personal learning style. No attempt was made to define the concept of “learning 
style”. Rather, the question was meant to provide the respondents with an opportunity to indicate whether or not the 
structure and organization of the WebIT program met their needs and expectations. The COMPLETERS were 
unanimous (11/11) in reporting the affirmative; the structure of the WebIT program seemed to suit them. The 
DROPPER-respondents were split with a simple majority (3/5) responding in the affirmative. Though in response to 
other questions, members of the DROPPER-respondent group offered numerous suggestions regarding ways to 
improve or alter the structure of WebIT to better address an individual need, the responses to this question indicate 
that the overall structure of the WebIT program was perceived to be satisfactory, even to many of those who were 
unable to successfully complete the program. One of the DROPPER-respondents who indicated that the WebIT 
design did not match his/her personal learning style provided the following elaborative comment: 

Overall, I believe in the philosophy that it should be up to the student to determine the best way to leam 
the material. I do not think, however, that this philosophy is suitable for an online curriculum. The lack of 
face to face [sic] contact was very difficult for me and I relied heavily on my cohort for support. Many 
times, we were all on /"different pages/". [«'c] While this was good in some ways (because the experience 
spectrum was very broad), I struggled a lot. I may be too /"old school/" [sic], but there were times when I 
just wanted to be told what to do and how to do it. I do learn by doing, but sometimes require (sadly) a little 
extra guidance. 
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Limitations 



The findings presented and discussed in this paper are based upon a very small sample of online program 
participants (N=16). All survey responses were both voluntary and anonymous, however it is impossible to ascertain 
the accuracy and truthfulness of individual responses, which is assumed. Additionally, it is impossible to determine 
whether or not the survey respondents' views and perspectives accurately reflect those of the larger population of 
individuals who might wish to pursue an online M.S. degree in Instructional Technology. Despite these limitations, 
several tentative conclusions can be drawn from this particular case context. However, any generalization of these 
conclusions to novel contexts should be made with caution. 

While it is impossible to know with certainty because of the small size of the sample being analyzed, it 
appears as though the DROPPER-respondents and the COMPLETERS differ in several meaningful ways that might 
help explain the high rate of attrition seen during the WebITl cohort. 

Conclusions and Recommendations 



Expectations and Preferences 

In terms of pre-program expectations, the DROPPER-respondents unanimously reported underestimating 
the amount of time and effort that would be required to successfully complete the program. While the majority of 
COMPLETERS also reported underestimating the time and effort that would be required (indicating that it is not 
simply the pre-program misconception that is critical) the key difference between these two groups may be found in 
their estimates of time spent each week engaged in completing program requirements. All of the 
DROPPER-respondents reported spending less than 15 hours per week on WebIT work, while most of the 
COMPLETERS reported spending more than 15 hours per week on WebIT work. Almost all (15/16) of the survey 
respondents underestimated the amount of time that would be needed to be successful in the WebIT program, but 
those who completed the program reported having spent more time engaged in program work. The critical 
difference between those survey respondents who completed the program and those who did not complete the 
program appears to be related to the individual's willingness and/or ability to allocate sufficient time to completing 
program requirements. Most likely, it is not the amount of time itself that is critical, but rather the ability to allocate 
the time necessary to be successful. 

Nearly all of the survey respondents indicated that the WebIT program was their third life -priority behind 
family and work. This is to be expected. However, when it comes to finding time to devote to academic pursuits, it 
is likely that such time can only be found by shifting the proportional allocations among life-priorities, and it is 
highly likely that those students who were most successful in doing this were more likely to complete the WebIT 
program. 

Several other differences between the two groups can be seen in their preferences for work pattern, 
program organization and pace. The DROPPER-respondents seem to prefer working alone to complete their 
academic work. They seem to prefer flexible schedules and timelines, and negotiating requirements with Instructors 
as opposed to being required to follow strict procedures and rigid timelines. Further, they seem to prefer being able 
to determine the pace at which they complete the academic program, rather than being required to work at a pre- 
determined pace, toward a specific end-point in time. The COMPLETERS seem to prefer a mix of independent and 
group work that is guided by deadlines and structure that is determined by the Instructor. They also seem to prefer a 
program structure that provides a set pace and fixed end-point in time. 

It is not clear that the preferences described above are critical in a student's decision to matriculate, 
continue or withdraw from an online program. However, it is clear that such programmatic characteristics can 
potentially constrain potential online students who intend to try to remain fully engaged in their pre-existing 
life -priorities while completing advanced graduate studies. It also is clear that it is difficult to design an instructional 
program that is simultaneously both structured and flexible; exclusively individualized and collaborative; an open- 
enrollment model and a cohort model. No particular programmatic structure is likely to be ideal for all learners. 

Recommendations Regarding Attrition 

Assuming that the differences between the COMPLETERS and DROPPER-respondents found in this study 
are real, there are several ways in which these might be addressed to reduce or minimize attrition. First, recruiting 
materials should clearly communicate to potential students a realistic appraisal of the time and effort that will likely 
be required of students for them to be successful in the program. Second, regardless of how the online academic 
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program is structured (open-enrollment, cohort model, individualized study model), some online students are likely 
to have time management difficulties of some type. If possible, programs should seek ways to accommodate such 
difficulties in order to provide a degree of individualized flexibility, as needed. One example might be a cohort 
program that would need to permit a student to withdraw from one cohort and re-enroll as a member of a subsequent 
cohort. Third, Instructors should be encouraged to be as flexible as possible in order to help students overcome time 
management difficulties during the completion of a class. Most online Instructors probably already do this to some 
extent, but any reasonable and relevant accommodations of this type might help retain students who encounter 
temporary difficulties. Fourth, online students often do not have access to the same support infrastructure as resident 
students. If possible, programs should attempt to overcome this shortcoming by providing a responsive human being 
to act in a contact/advisor/support role. Not only will this lessen the burden on Instructors, it can help to ensure that 
the nature of the support service is of consistently high quality. Timely interactions, especially of a problem-solving 
nature, are often critical to an online student's success and can help alleviate a student's difficulties. 
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Purpose 

The current case study represents an evaluation of a workshop designed and developed to help 
undergraduate college students learn the basics of using the virtual world Second Life, developed by Linden Labs, in 
education. The workshop took place in 2010 and included both students and teachers. In this paper are described 
briefly the quantitative and qualitative evaluation findings of the prototype implementation of the workshop. 

Background 

Second Life was launched in 2003 as a virtual world in which users create avatars and interact with one 
another in a virtual setting (Baker, Wentz, & Woods, 2009). Second Life users can manipulate objects, create 
objects, and alter their appearance. The virtual world SL is made up of many islands where avatars construct objects, 
meet, and spend virtual dollars (Linden dollars. As a participant in SL, a user has a choice of a basic or premium 
account. The basic account is free, while the premium account is $9.95 a month. An institution or instructor might 
opt for the premium account, because of the ability to own land. Land can be purchased from Linden Labs and set 
up to a users specification. Additionally, the landowner can build objects for use on the land (Rymaszewski, et al., 
2008). 



Many of the studies cited in the literature review feature a short tutorial on how to use SL; the tutorials 
focus on navigating, communication, and editing appearance. Additionally, researchers often build and script their 
own objects. Complex objects can be built in SL, by utilizing the available prims; furthermore, by using Linden 
scripting language objects can perform tasks (Rymaszewski, et al. 2008). 

Baker, S.C., Wente, R.K., and Woods, M. M. (2009) noted that it is important to have a clear objective for 
instruction in SL, and furthermore the instructor needs to iterate why SL is the right environment to use for the 
objective. When first introducing SL to a class it is necessary to start with a simple introductory exercise, such as 
having students explores different locations and then writing assignments. In addition, research into collaboration in 
SL has noted several important steps: delivery of the material, order of assignments, and the start time (Lucia, A. D., 
Francese, R., Passero, I., and Tortora, G. (2009). SL is featured prominently in cases that exploit it’s collaboration 
features (Edirisingha, et al., 2009; Jarmon, Traphagan, Mayrath, & Trivedi, 2009; King., Greidanus, Carbonaro, 
Drummond, & Patterson, 2009; Lucia, Francese, Passero, & Tortora, 2009). 
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Lucia, A. D., Francese, R., Passero, I., and Tortora, G. (2009) constructed a SL campus for experiments 
pertaining to group interactions. The authors first created small objects in what are called sandboxes. Once the 
researchers were able to successfully build objects and script actions, an island was purchased in SL, and the 
researchers set up a virtual campus. 

There have been several evaluations pertaining to Second Life and education; most of these evaluations 
discuss the strengths, and pitfalls of instruction in a virtual environment. Baker, S., C., Wente, R. K., and Woods, 
M.M. (2009) noted that one potential drawback of SL is the investment of learning time, however students were able 
to get up and running within an hour of first stating SL. Additionally Baker, S.C., Wente, R.K., and Woods, M. M. 
(2009) find that in order to successfully utilize SL, instructors need to employ new classroom management 
techniques. The authors note an example of a class discussion taking too long due to delays in responses. 
Researchers, Edirisingha, P., Nie, M., Pluciennik, M., and Young, R. (2009) provided students with a training 
session before implementing a study to measure the effects of socialization in SL. Edrisingha, P., Nie, M., 
Pluciennik, M., and Young, R. (2009) concluded that successful training of tutors in technical SL, and peer 
moderation took 3 hours to complete. The researchers trained participants through the Media Zoo in SL, and relied 
on text chats. Additionally, participants were provided a 12-page training guide that was produced by Wheeler and 
Salmon, 2008, and given several links to training videos. 



In the research of Mayrath, M., Sanchez, J., Traphagan, T., Heiks, J., and Tavedi, A. (2007) a pilot study 
was used to understand how to effectively use SL for instruction. In the pilot study the researchers were concerned 
with leaner gains, motivation, technical skills needed, and usability. The pilot study was conducted to measure the 
effectiveness of SL, and goal of the researchers was to use SL to teach a world literature and rhetoric course The 
study utilized several exercises within SL, during the first semester students were instructed to build an online 
campus in SL, however in the second semester students were instructed to create an avatar after a role model. 

During the course of the study, Mayrath, et al. (2007) collected surveys, interviewed students, and made 
observations that were concerned with self-efficacy, the students use of SL, and the students attitude toward SL. 

Through the interviews, the researchers concluded that the building activity was overwhelming to the 
students; therefore during the second semester researchers utilized the role model activity. Eighty-two percent of the 
participants felt that the role modeling activity offered positive social interaction and 61% agreed that the ability to 
see an avatar during a chat made for a better discussion. (Mayrath, et al., 2007) The use of SL in education has been 
explored in different learning domains and has been used in conjunction with learning models. 

Schiller, S. Z. (2009) applied the learner-centered teaching methodology and the five concepts of learner 
centered teaching, explored by Weimer, (2002), to SL. Schiller, S. Z., (2009) implemented SL in a MBA course, the 
purpose of using SL was to learn the meaning of virtual business, show specific examples of virtual businesses, and 
to develop socialization skills. As in the study by Mayrath et al. (2007), Schiller, S. Z. (2009) required time for 
students to become familiar with SL. Herold, D. K. (2010), conducted a similar study, in which the learning domain 
communications and the course was titled “Media Studies and Everyday Life”. Harold, D. K. (2010) used SL to 
demonstrate how representations can convey a message, for instance, the headquarters of CNN is much like an open 
campus, whereas Reuters has a more corporate structure. 

Studies relating to the subject of SL often cite learner’s attitudes, motives, and familiarity with technology 
as a qualitative measure. For instance, Schiller, S. Z. (2009) found that the average student (mean=3.26, s=0.25) felt 
comfortable to completing the main SL project successfully. Furthermore, an average amount of students 
(mean=3.44, s=0.24) had fun completing the SL project. However, the researchers found that students were not 
comfortable moving avatars around in SL (mean=2.74, s=0.22). Lucia, A.D., Francese, R., Passero, I., and Tortora, 
G. (2009) found that students can develop a sense of control in SL. Additionally, these authors concluded that 
students felt it was easier to communicate through discussions with other avatars. 

The purpose of the present study was to evaluate the effectiveness of a workshop about using the virtual 
world Second Life for educational purposes. Specifically, it collected evidence of effectiveness from college 
undergraduate students in terms of their successful completion of activities within the Second Life environment and 
attitudinal change (e.g., willingness to use Second Life in classroom in the future) after the workshop. 
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Methods 

Participants and approach 

A total of 22 individuals (11 females and 9 males) from a large southwestern university in the US 
participated in the study. They were recruited primarily from an undergraduate introductory computer literacy 
course. The participants included four freshmen, four sophomore, four juniors, four graduate students and four 
employees of the university. Those who were students received course credit for their participation; employees 
participated out of interest. 

Due to technical problems, two participants’ post survey data were missing. Therefore, they were excluded, 
leaving 20 participants for the analysis. Among them, one participant had one missing data point, with the rest of the 
data being complete. Mean imputation was used to estimate this missing data point. 

This study employed a case study approach to a design, development and evaluation project. Pre-surveys 
were administered before the workshop and post-surveys as well as a posttest were administered after the workshop. 

Procedures 

The instructional workshop was conducted as a 90-minute lesson in a classroom setting with 35 Dell 
desktop computers. Participants completed three objectives within the Second Life environment; (a) Objective 1: To 
successfully search, navigate, and save specified locations within Second Life; (b) Objective 2: To communicate 
effectively in Second Life using different methods (e.g. microphone and keyboard) and (c). Objective 3: To 
effectively describe different ways to incorporate Second Life into an educational environment. 

The instructor introduced the workshop on Second Life and how to create an avatar. The instructor 
demonstrated how to move with an avatar and summarized the interface. The participants completed several 
embedded exercises, such as finding a specific location. After completing the embedded exercises, the instructor 
demonstrated communication in Second Life. The participants were required to send an instant message to the 
instructor. To introduce the third objective, the instructor demonstrated how to find educational specific islands. 

Materials and environment 

The computer-based environment was the Second Life virtual world. The environment provided 
information about whether each participant completed or failed to complete to the activities specified in each of the 
three workshop objectives. The paper-pencil materials used for the workshop included a student participant guide 
(29 pages), and an instructor guide (28 pages). 

Measures and instruments 

At the end of the workshop participants completed a set of posttest activities to determine participants’ 
achievement of the three objectives (see Table 1). 

Students completed a pre-survey at the beginning of the workshop that included a few demographic 
questions (gender and class level) and four attitude questions. These four questions (see Table 1) were completed 
using a 5-point Likert-type scale ranging from “1” ( strongly agree ) to “5” ( strongly disagree). The Cronbach’s alpha 
of these four questions was .92. This pre-survey was completed by each of the participants before the introduction to 
Second Life. Finally, participants completed a post-survey, similar to the pre-survey (see Table 2; this measure 
included five subjective questions that employed the same 5-point Likert-type scale. 

In addition to the quantitative measures, participants were also asked to answer several open-ended 
questions (e.g., “ What content would be appropriate for a class that takes place in Second Life?”). 
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Results 



Participants successfully completed 55%, 80% and 90% of the activities specified in Objective 1, 2 and 3 
respectively. 

Table 2 presents the descriptive statistics for the attitude questions. A one-sample chi-square test was 
conducted to assess whether more participants chose to agree or strongly agree that they were willing to use Second 
Life in their classroom after the instruction. The results were significant,/ 2 (2, N = 20) = 16.60, p < .001, indicating 
that participants were positive about using Second Life in classroom settings. Qualitative data provided further 
evidence. For instance, one participant said, “I am interested in teaching training for health related content.” A 
second one-sample chi-square test was conducted to assess whether more participants chose to agree or strongly 
agree that they were likely to use Second Life in their classroom after the instruction. The results were significant, % 
(2, N =20)= 15.35, p < .001. This indicated that participants believed Second Life has the potential to be used in the 
classroom. The possible reason from the qualitative data may be that “most people have an idea how to operate even 
the most basic computer workshops.” The third test was conducted to assess whether more participants chose to 
agree or strongly agree that they were willing to take a semester-long class on Second Life after the instruction. The 
results were non-significant, x (2, N = 20) = 4. 10, p = .13. The last test was conducted to assess whether more 
participants chose to agree or strongly agree that they were willing to take additional classes about using Second 
Life after the instruction. The results were non-significant, x (2, N = 20) = 1.87, p = .39. 

The qualitative data were comprised of responses to 10 questions in the pretest and posttest. The questions 
focused on how the participant could use SL in the classroom, how the participant felt about the workshop, and how 
were their experience using SL. For instance, in response to the question, “What if any are the biggest challenges to 
using SL in the classroom, 6 participants cited classroom management issues. Flowever, 6 participants responded 
positively that they would use SL again. One participant noted that, “it is a great engagement activity and the new 
college students are digital natives”. Additionally, the qualitative data explored the possibilities of using SL in the 
classroom. 

In several questions, participants were asked how they would use SL in the classroom. For example, 6 
participants noted that SL would be an appropriate tool for a general studies course, while 5 participants cited that 
SL would provide a “virtual field trip”. In conjunction with this question, 16 participants left the workshop more 
interested in using SL. Flowever, several participants cited several impediments to implementing SL. 

The surveys included several questions regarding challenges and the time investment necessary to use SL 
in the classroom. The participants noted that classroom management would be a challenge; one participant wrote, “it 
would be hard to watch over all the students to see if they were doing what they should be on Second Life”. A major 
concern for all the participants was keeping focus and keeping students on task. Although the participants expressed 
an interest in using SL again, 10 participants expressed doubts. One participant wrote, “not much, because 1 think it 
is better to be active and learn in real life”. Some participants required more exploration in SL before coming to a 
conclusion. 

We also compared participants’ qualitative responses before and after the instructional workshop and 
results showed positive attitudinal change. For example, when describing challenges in using Second Life 
instractionally, in the pre survey, one participant described “accessing and learning curve”. In the post-survey, the 
same participant said, “none, it’s very simple to use”. In the post-survey data, nine of the participants suggested they 
would use Second Life to support visual activities. For instance, one participant described, “visually representing 
molecules in chemistry, at museums, historic places (field trips)”. 

We found that 19 out of 20 participants agreed or strongly agreed that they felt comfortable using Second 
Life, ( Mean = 4.3, SD = .92). We computed correlation coefficients between the perceived comfort of using Second 
Life and (a) willingness of using Second Life in classroom, (b) willingness of taking a semester long class on 
Second Life, (c) likelihood of using Second Life in classroom and (d) likelihood of taking additional class on 
Second Life. We found that students’ perceived levels of comfort after taking the workshop was significantly 
correlated with their willingness to take a semester-long class (r = .62 ,p = .003), the likelihood of their using 
Second Life in the classroom (/' = .54,/? = .01) and their likelihood of taking an additional class on it (/' = .58,/? =. 
007) 
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Discussion 



The purpose of this study was to evaluate the effectiveness of an instructional workshop designed to teach 
how to use Second Life in an educational setting. The results showed that the majority of participants successfully 
completed activities involving searching, communicating and using educational features within the Second Life 
virtual environment. In some studies, it was found that students might feel frustrated in a virtual world (e.g., Barab, - 
et al, 2007). However, in this study we found that participants felt comfortable with Second Life. In addition, 
participants indicated their willingness to further use Second Life in a classroom setting and this willingness was 
associated with their perceived levels of comfort. These findings indicated that there is good potential to implement 
Second Life in classrooms to facilitate students’ communication as well as their engagement in learning. For 
instance, Second Life could be used in an introductory teaching-with-technology workshop for teachers interested in 
virtual reality. Further investigations regarding using Second Life in classrooms appear to be warranted. 
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Table 1 

Posttest questions and results 



Post-test question 


Mean 


Standard Deviation 


Navigate to the Second Life Louvre, Harvard University, 
The United States White House, and Open University, 
save their location in your favorites. 


0.55 


0.51 


Find three strangers and meet them in Second Life. You 
can search for people or find people in the virtual 
environment. Make at lease 3 connections. 


0.45 


0.51 



Friendship each of the participants and the instructor in the 0.25 0.44 

workshop 



Demonstrate how to use the chat feature by sending a 0.70 0.47 

instant message and communicating in the open chat box. 



Table 2 



Descriptive statistics for attitude items on the pre-survey and post-survey 



Pre-survey Items 


Mean (SD) 


Post-survey Items 


Mean (SD) 


I am willing to use Second Life in 
my classroom. 


3.50 (.95) 


I am willing to use Second Life in 
my classroom. 


3.80 (1.06) 


I am willing to take a semester long 
class on Second Life. 


2.85 (1.14) 


I am willing to take a semester long 
class on Second Life. 


3.30 (1.30) 


I am likely to use Second Life in the 
classroom. 


2.9 (.91) 


I am likely to use Second Life in the 
classroom. 


3.50 (1.05) 


I am likely to take additional classes 
about Second Life in the classroom. 


3.10(1.02) 


I am likely to take additional classes 
about Second Life in the classroom. 
After this workshop I feel more 
comfortable using Second Life. 


3.20(1.15) 
4.30 (.92) 
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Using Second Life for Educational Purposes: An Evaluation of a New Workshop 

Kyle M. Wright 
Wilhelmina C. Savenye 
Arizona State University 

Purpose 

The current case study represents an evaluation of a workshop designed and developed to help 
undergraduate college students learn the basics of using the virtual world Second Life, developed by Linden Labs, in 
education. The workshop took place in 2010 and included both students and teachers. In this paper are described 
briefly the quantitative and qualitative evaluation findings of the prototype implementation of the workshop. 

Background 

Second Life was launched in 2003 as a virtual world in which users create avatars and interact with one 
another in a virtual setting (Baker, Wentz, & Woods, 2009). Second Life users can manipulate objects, create 
objects, and alter their appearance. The virtual world SL is made up of many islands where avatars construct objects, 
meet, and spend virtual dollars (Linden dollars. As a participant in SL, a user has a choice of a basic or premium 
account. The basic account is free, while the premium account is $9.95 a month. An institution or instructor might 
opt for the premium account, because of the ability to own land. Land can be purchased from Linden Labs and set 
up to a users specification. Additionally, the landowner can build objects for use on the land (Rymaszewski, et al., 
2008). 



Many of the studies cited in the literature review feature a short tutorial on how to use SL; the tutorials 
focus on navigating, communication, and editing appearance. Additionally, researchers often build and script their 
own objects. Complex objects can be built in SL, by utilizing the available prims; furthermore, by using Linden 
scripting language objects can perform tasks (Rymaszewski, et al. 2008). 

Baker, S.C., Wente, R.K., and Woods, M. M. (2009) noted that it is important to have a clear objective for 
instruction in SL, and furthermore the instructor needs to iterate why SL is the right environment to use for the 
objective. When first introducing SL to a class it is necessary to start with a simple introductory exercise, such as 
having students explores different locations and then writing assignments. In addition, research into collaboration in 
SL has noted several important steps: delivery of the material, order of assignments, and the start time (Lucia, A. D., 
Francese, R., Passero, 1., and Tortora, G. (2009). SL is featured prominently in cases that exploit it’s collaboration 
features (Edirisingha, et al., 2009; Jarmon, Traphagan, Mayrath, & Trivedi, 2009; King., Greidanus, Carbonaro, 
Drummond, & Patterson, 2009; Lucia, Francese, Passero, & Tortora, 2009). 

Lucia, A. D., Francese, R., Passero, I., and Tortora, G. (2009) constructed a SL campus for experiments 
pertaining to group interactions. The authors first created small objects in what are called sandboxes. Once the 
researchers were able to successfully build objects and script actions, an island was purchased in SL, and the 
researchers set up a virtual campus. 

There have been several evaluations pertaining to Second Life and education; most of these evaluations 
discuss the strengths, and pitfalls of instruction in a virtual environment. Baker, S., C., Wente, R. K., and Woods, 
M.M. (2009) noted that one potential drawback of SL is the investment of learning time, however students were able 
to get up and running within an hour of first stating SL. Additionally Baker, S.C., Wente, R.K., and Woods, M. M. 
(2009) find that in order to successfully utilize SL, instructors need to employ new classroom management 
techniques. The authors note an example of a class discussion taking too long due to delays in responses. 
Researchers, Edirisingha, P., Nie, M., Pluciennik, M., and Young, R. (2009) provided students with a training 
session before implementing a study to measure the effects of socialization in SL. Edrisingha, P., Nie, M,, 
Pluciennik, M., and Young, R. (2009) concluded that successful training of tutors in technical SL, and peer 
moderation took 3 hours to complete. The researchers trained participants through the Media Zoo in SL, and relied 
on text chats. Additionally, participants were provided a 12-page training guide that was produced by Wheeler and 
Salmon, 2008, and given several links to training videos. 
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In the research of Mayrath, M., Sanchez, J., Traphagan, T., Heiks, J., and Tavedi, A. (2007) a pilot study 
was used to understand how to effectively use SL for instruction. In the pilot study the researchers were concerned 
with leaner gains, motivation, technical skills needed, and usability. The pilot study was conducted to measure the 
effectiveness of SL, and goal of the researchers was to use SL to teach a world literature and rhetoric course The 
study utilized several exercises within SL, during the first semester students were instructed to build an online 
campus in SL, however in the second semester students were instructed to create an avatar after a role model. 

During the course of the study, Mayrath, et al. (2007) collected surveys, interviewed students, and made 
observations that were concerned with self-efficacy, the students use of SL, and the students attitude toward SL. 

Through the interviews, the researchers concluded that the building activity was overwhelming to the 
students; therefore during the second semester researchers utilized the role model activity. Eighty-two percent of the 
participants felt that the role modeling activity offered positive social interaction and 61% agreed that the ability to 
see an avatar during a chat made for a better discussion. (Mayrath, et al., 2007) The use of SL in education has been 
explored in different learning domains and has been used in conjunction with learning models. 

Schiller, S. Z. (2009) applied the learner-centered teaching methodology and the five concepts of learner 
centered teaching, explored by Weimer, (2002), to SL. Schiller, S. Z., (2009) implemented SL in a MBA course, the 
purpose of using SL was to learn the meaning of virtual business, show specific examples of virtual businesses, and 
to develop socialization skills. As in the study by Mayrath et al. (2007), Schiller, S. Z. (2009) required time for 
students to become familiar with SL. Herold, D. K. (2010), conducted a similar study, in which the learning domain 
communications and the course was titled “Media Studies and Everyday Life”. Harold, D. K. (2010) used SL to 
demonstrate how representations can convey a message, for instance, the headquarters of CNN is much like an open 
campus, whereas Reuters has a more corporate structure. 

Studies relating to the subject of SL often cite learner’s attitudes, motives, and familiarity with technology 
as a qualitative measure. For instance, Schiller, S. Z. (2009) found that the average student (mean=3.26, s=0.25) felt 
comfortable to completing the main SL project successfully. Furthermore, an average amount of students 
(mean=3.44, s=0.24) had fun completing the SL project. However, the researchers found that students were not 
comfortable moving avatars around in SL (mean=2.74, s=0.22). Lucia, A.D., Francese, R., Passero, I., and Tortora, 
G. (2009) found that students can develop a sense of control in SL. Additionally, these authors concluded that 
students felt it was easier to communicate through discussions with other avatars. 

The purpose of the present study was to evaluate the effectiveness of a workshop about using the virtual 
world Second Life for educational purposes. Specifically, it collected evidence of effectiveness from college 
undergraduate students in terms of their successful completion of activities within the Second Life environment and 
attitudinal change (e.g., willingness to use Second Life in classroom in the future) after the workshop. 

Methods 

Participants and approach 

A total of 22 individuals (11 females and 9 males) from a large southwestern university in the US 
participated in the study. They were recruited primarily from an undergraduate introductory computer 
literacy course. The participants included four freshmen, four sophomore, four juniors, four graduate 
students and four employees of the university. Those who were students received course credit for their 
participation; employees participated out of interest. 

Due to technical problems, two participants’ post survey data were missing. Therefore, they were 
excluded, leaving 20 participants for the analysis. Among them, one participant had one missing data point, 
with the rest of the data being complete. Mean imputation was used to estimate this missing data point. 

This study employed a case study approach to a design, development and evaluation project. Pre-surveys 
were administered before the workshop and post-surveys as well as a posttest were administered after the workshop. 

Procedures 
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The instructional workshop was conducted as a 90-minute lesson in a classroom setting with 35 Dell 
desktop computers. Participants completed three objectives within the Second Life environment; (a) 
Objective 1: To successfully search, navigate, and save specified locations within Second Life; (bj Objective 2: 
To communicate effectively in Second Life using different methods (e.g. microphone and keyboard] and (c). 
Objective 3: To effectively describe different ways to incorporate Second Life into an educational 
environment. 

The instructor introduced the workshop on Second Life and how to create an avatar. The instructor 
demonstrated how to move with an avatar and summarized the interface. The participants completed several 
embedded exercises, such as finding a specific location. After completing the embedded exercises, the instructor 
demonstrated communication in Second Life. The participants were required to send an instant message to the 
instructor. To introduce the third objective, the instructor demonstrated how to find educational specific islands. 

Materials and environment 

The computer-based environment was the Second Life virtual world. The environment provided 
information about whether each participant completed or failed to complete to the activities specified in each of the 
three workshop objectives. The paper- pencil materials used for the workshop included a student participant guide 
(29 pages), and an instructor guide (28 pages). 

Measures and instruments 

At the end of the workshop participants completed a set of posttest activities to determine participants’ 
achievement of the three objectives (see Table 1 ). 

Students completed a pre-survey at the beginning of the workshop that included a few demographic 
questions (gender and class level) and four attitude questions. These four questions (see Table 1) were completed 
using a 5-point Likert-type scale ranging from “1” ( strongly agree ) to “5” ( strongly disagree). The Cronbach’s alpha 
of these four questions was .92. This pre-survey was completed by each of the participants before the introduction to 
Second Life. Finally, participants completed a post-survey, similar to the pre-survey (see Table 2; this measure 
included five subjective questions that employed the same 5-point Likert-type scale. 

In addition to the quantitative measures, participants were also asked to answer several open-ended 
questions (e.g., “ What content would be appropriate for a class that takes place in Second Life?”). 

Results 

Participants successfully completed 55%, 80% and 90% of the activities specified in Objective 1, 2 and 3 
respectively. 

Table 2 presents the descriptive statistics for the attitude questions. A one-sample chi-square test was 
conducted to assess whether more participants chose to agree or strongly agree that they were willing to use Second 
Life in their classroom after the instruction. The results were significant,/ 2 (2, N = 20) = 16.60, p < .001, indicating 
that participants were positive about using Second Life in classroom settings. Qualitative data provided further 
evidence. For instance, one participant said, “I am interested in teaching training for health related content.” A 
second one-sample chi-square test was conducted to assess whether more participants chose to agree or strongly 
agree that they were likely to use Second Life in their classroom after the instruction. The results were significant, y 2 
(2, N = 20) = 15.35, p < .00 1 . This indicated that participants believed Second Life has the potential to be used in the 
classroom. The possible reason from the qualitative data may be that “most people have an idea how to operate even 
the most basic computer workshops.” The third test was conducted to assess whether more participants chose to 
agree or strongly agree that they were willing to take a semester-long class on Second Life after the instruction. The 
results were non-significant, x 2 (2, N = 20) - 4.10, p - .13. The last test was conducted to assess whether more 
participants chose to agree or strongly agree that they were willing to take additional classes about using Second 
Life after the instruction. The results were non-significant, % 2 (2, N = 20) = 1.87, p = .39. 
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The qualitative data were comprised of responses to 10 questions in the pretest and posttest. The questions 
focused on how the participant could use SL in the classroom, how the participant felt about the workshop, and how 
were their experience using SL. For instance, in response to the question, “What if any are the biggest challenges to 
using SL in the classroom, 6 participants cited classroom management issues. However, 6 participants responded 
positively that they would use SL again. One participant noted that, “it is a great engagement activity and the new 
college students are digital natives”. Additionally, the qualitative data explored the possibilities of using SL in the 
classroom. 

In several questions, participants were asked how they would use SL in the classroom. For example, 6 
participants noted that SL would be an appropriate tool for a general studies course, while 5 participants cited that 
SL would provide a “virtual field trip”. In conjunction with this question, 16 participants left the workshop more 
interested in using SL. However, several participants cited several impediments to implementing SL. 

The surveys included several questions regarding challenges and the time investment necessary to use SL 
in the classroom. The participants noted that classroom management would be a challenge; one participant wrote, “it 
would be hard to watch over all the students to see if they were doing what they should be on Second Life”. A major 
concern for all the participants was keeping focus and keeping students on task. Although the participants expressed 
an interest in using SL again, 10 participants expressed doubts. One participant wrote, “not much, because I think it 
is better to be active and learn in real life”. Some participants required more exploration in SL before coming to a 
conclusion. 

We also compared participants’ qualitative responses before and after the instructional workshop and 
results showed positive attitudinal change. For example, when describing challenges in using Second Life 
instructionally, in the pre survey, one participant described “accessing and learning curve”. In the post-survey, the 
same participant said, “none, it’s very simple to use”. In the post-survey data, nine of the participants suggested they 
would use Second Life to support visual activities. For instance, one participant described, “visually representing 
molecules in chemistry, at museums, historic places (field trips)”. 

We found that 19 out of 20 participants agreed or strongly agreed that they felt comfortable using Second 
Life, ( Mean = 4.3, SD = .92). We computed correlation coefficients between the perceived comfort of using Second 
Life and (a) willingness of using Second Life in classroom, (b) willingness of taking a semester long class on 
Second Life, (c) likelihood of using Second Life in classroom and (d) likelihood of taking additional class on 
Second Life. We found that students’ perceived levels of comfort after taking the workshop was significantly 
correlated with their willingness to take a semester-long class ( r = .62, p = .003), the likelihood of their using 
Second Life in the classroom (r = .54,/? = .01) and their likelihood of taking an additional class on it (r = .58,/? =. 
007) 



Discussion 

The purpose of this study was to evaluate the effectiveness of an instructional workshop designed to teach 
how to use Second Life in an educational setting. The results showed that the majority of participants successfully 
completed activities involving searching, communicating and using educational features within the Second Life 
virtual environment. In some studies, it was found that students might feel frustrated in a virtual world (e.g., Barab, - 
et al, 2007). However, in this study we found that participants felt comfortable with Second Life. In addition, 
participants indicated their willingness to further use Second Life in a classroom setting and this willingness was 
associated with their perceived levels of comfort. These findings indicated that there is good potential to implement 
Second Life in classrooms to facilitate students’ communication as well as their engagement in learning. For 
instance. Second Life could be used in an introductory teaching-with-technology workshop for teachers interested in 
virtual reality. Further investigations regarding using Second Life in classrooms appear to be warranted. 
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Table 1 

Posttest questions and results 



Post-test question Mean Standard Deviation 



Navigate to the Second Life Louvre, Harvard University, 
The United States White House, and Open University, 
save their location in your favorites. 


0.55 


0.51 


Find three strangers and meet them in Second Life. You 
can search for people or find people in the virtual 
environment. Make at lease 3 connections. 


0.45 


0.51 


Friendship each of the participants and the instructor in the 0.25 

workshop 


0.44 


Demonstrate how to use the chat feature by sending a 
instant message and communicating in the open chat box. 


0.70 


0.47 


Table 2 

Descriptive statistics for attitude items on the pre-survey and post-survey 




Pre-survey Items Mean (SD) 

I am willing to use Second Life in 3.50 (.95) 

my classroom. 

I am willing to take a semester long 2.85 (1.14) 

class on Second Life. 

I am likely to use Second Life in the 2.9 (.91) 
classroom. 

I am likely to take additional classes 3.10 (1 .02) 
about Second Life in the classroom. 


Post-survey Items 

I am willing to use Second Life in 
my classroom. 

I am willing to take a semester long 
class on Second Life. 

I am likely to use Second Life in the 
classroom. 

I am likely to take additional classes 
about Second Life in the classroom. 
After this workshop I feel more 
comfortable using Second Life. 


Mean (SD) 

3.80 (1.06) 

3.30 (1.30) 
3.50 (1.05) 
3.20(1.15) 

4.30 (.92) 
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Abstract: This study was conducted to investigate the acceptance levels of employees towards e-learning in the 
workplace of South Korea. Two hundred and sixty one employees of a food service company voluntarily 
participated in the study in Fall 2010 and 170 participants’ data were valid for final analyses. The survey consisted 
of 33 items drawn from selected categories from the Unified Theory of Acceptance and Use of Technology 
(UTAUT). The results indicated significant differences in the acceptance levels towards e-learning based on gender, 
age, position, work experience, and e-learning experience. Interestingly, locations of employees did not affect the 
differences in the acceptance levels towards e-learning. The results showed that performance expectancy, attitudes, 
and anxiety were the greatest factors that influenced employees’ intention to use e-learning in the workplace. These 
findings suggest that Human Resource Development (HRD) staff should consider factors of technology acceptance 
in the design and selection of e-learning in the workplace. The limitation of the study includes a small sample size 
and direct and indirect factors influencing participants’ response such as limited technical infrastructure. 

Introduction 

In past years, employees were able to sufficiently carry out their jobs with knowledge they acquired in 
schools because the change in new knowledge was intermittent. Nowadays, employees in the workplace need to 
learn to keep up with new information or skills to do their jobs (Gunasekaran, McNeil, & Shaul, 2002). Hence, 
many companies are being driven to seek new ways to train their employees in the workplace, and to keep 
developing them. 

E-learning is considered to be the instructional use of technology and includes teaching and learning 
(Masters & Eliaway, 2008; Rosenberg, 2001). E-learning has been recognized as a new way of developing 
employees and it allows organizations to increase competitiveness without the high cost of travel and time away 
from work (Bates, 2005). General Motors, Motorola, and Accenture are examples of corporations that have moved 
some of their training into e-learning (Bates, 2005). 

These trends, in turn, have combined to support the recent growth of e-learning. According to Rosset & 
Marshall (2010), nearly one-third of training content is delivered through e-learning. However, as companies’ 
investment in e-learning grows, so do questions about its value to business. 

The needs of this growing population of employees and the number of organizations that have adopted 
e-learning require more research. Although there are numerous studies on the topic of e-learning, most research was 
conducted in academic settings, leaving a gap in the research on e-learning in the workplace (Ingram, Biermann, 
Cannon, Neil & Waddle, 2000; Keller & Cernerud, 2002; Liaw, Huang, & Chen, 2007; Link & Marz, 2006). 
Furthermore, not all organizations have been successful at implementing e-leaming (Resenberg, 2001). 

According to Keil (1995), even though computer technology has become pervasive in the workplace, work 
productivity is lower than expected due to poor user technology acceptance. When users utilize any technology 
including e-learning, their acceptance of the technology is the most important determinant of their intention to 
continually use the technology (Roca & Gagne, 2008). Therefore, understanding employees’ acceptance towards 
e-learning is important to improving e-learning usage in the workplace. This study intended to answer the following 
research questions. 

• What are employees’ acceptance levels towards e-learning in the workplace? 

• Is there a significant difference in the acceptance levels of employees towards e-learning based on 

gender, age, work experience, position, workplace location and E-learning experience? 

• Which factors affect employees’ intention to use e-learning in the workplace? 
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E-learning in the workplace 

Many organizations have utilized e-learning as their training method due to the effectiveness of e-learning. 
The effectiveness of e-learning includes both better service and low cost (Bates, 2010). E-learning is getting popular 
in the workplace due to its advantages. First, many workplaces ask employees to keep up their knowledge as well as 
skills for their job and e-learning can provide employees with flexible access to training materials (Jia, Wang, Ran, 
Yang, Liao, & Chiu, 2011). In addition, e-learning offers organizations many benefits, such as worldwide updates, 
instant delivery of training materials in various formats, consistent quality, and cost effectiveness (Biech, 2008). 
Second, educational technologies, including e-learning enable companies to offer personalized learning in order to 
adapt to different learners’ needs and equal opportunities (Bates, 2010; Biech, 2008). Organizations can offer 
learning chances and support to any employees in the workplace, regardless of the location of the workplace, gender, 
or cultural differences (Biech, 2008). 

Nearly 85 % of Fortune 500 companies utilize e-learning for developing their employees’ knowledge and 
skills (Barron, 2003). Organizations invested over 250 billion dollars for training, of which 16 billion dollars were 
spent on e-learning training (Johnson, Hornik, & Salas, 2008). Additionally, the e-learning portion of employee 
training shows drastic growth from 15.4% in 2002, and 36.5 % in 2009 (ASTD, 2010). While the rapid growth of 
e-learning has been demonstrated in the U. S., South Korea has also been following the e-learning growth trend. 

The development of e-leaming in South Korea is strongly related to the rapid growth of its information and 
communications technology industry (Misko, Choi, Hong & Lee, 2005). According to Korean Research Institution 
for Vocational Education and Training (KRIVET), the numbers of companies and employees who have participated 
in e-learning courses in the workplace have rapidly increased since 1999. In 1999, the number of employees who 
participated in e-learning was only 19, 653, but the total number of participants in 2005 was 1, 061, 985, a 54% 
increase in six years. E-learning training comprised over 45% of total training in 2005 (Byun, & Lee, 2007; Lee, 
Byun, Kwon, & Kwak, 2008). E-leaming has become a universal training method for the workplace in South Korea 
(Lee, et al., 2008). 

Technology Acceptance and e-learning 

Although many organizations have utilized e-learning to improve their performance due to the many 
benefits of e-learning, the value of e-learning may not be fully appreciated unless users accept e-learning as a 
learning tool (Lee, Yoon, & Lee, 2009). To explore users’ acceptance and utilization of e-learning, the United 
Theory of Acceptance and Use of Technology (UTAUT) was used. UTAUT, a more recent instalment, was used to 
examine how people adopt and use e-learning (Masrom, 2007; Selim, 2003). 

The UTAUT model has been applied to examine research that is related to both academic settings and the 
workplace. UTAUT integrates elements of the following: Theory of Reasoned Action, Motivation Model, Theory of 
Planned Behavior (TPD), Technology acceptance Model (TAM), a combined TAM and TPB model, Model of PC 
utilization. Innovation Diffusion Theory, and Social Cognition Theory (Venkatesh, Morris, & Davis, 2003). 

UTAUT by Venkatesh et al., (2003) was empirically tested with original data from the four organizations 
and then cross-validated using new data from an additional two organizations. These tests provided strong empirical 
support for UTAUT, which revealed three direct determinants of intention to use (performance expectancy, and 
social influence) and two direct determinants of usage behavior (intention and facilitating conditions). In addition, 
UTAUT was validated in cross-cultural settings (Oshlyansky, Cairns, & Thimbleby, 2007). The researchers 
validated UTAUT through conducting research in the Czech Republic, Greece, India, Malaysia, New Zealand, Soudi 
Arabia, South Africa, the United Kingdom, and the United States (Oshlyansky et al., 2007). 

Recent studies focused on a wide variety of factors that affect users’ acceptance of e-learning (Liaw et al, 
2007; Oshlyansky et al., 2007). Therefore, understanding individual’s attitudes towards e-learning facilitates the 
creation of appropriate e-learning environments. 



Method 

The study was conducted through a survey in 2010 in a food service company in South Korea. 

For analyses, some of the survey data were deleted due to incompleteness and errors. This left 170 surveys and all 
data were collected via voluntary participations by a web-based survey interface. Factor analysis was conducted to 
reduce variables and regression analysis was examined to investigate which factors affect the intention to use 
e-learning in the workplace. In addition, t-test and ANOVA were conducted to examine the differences of 
employees’ acceptance towards e-learning based on gender, age, position, work experience, workplace location and 
e-leaming experience. 
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Measuring E-learning Acceptance Level 

This research team targeted employees’ acceptance levels of E-learning in the workplace. To investigate 
employees’ utilization levels of E-learning, the research team created surveys based on selected items from the 
Unified Theory of Acceptance and Use of Technology’ (UTAUT) (Venkatesh et al., 2003) by using a 7-point Likert 
Scale in the following categories: Performance expectancy, effort expectancy, attitude, social influence, facilitating 
condition, anxiety and the intention to use e-learning. See Table 1 for survey items on E-learning utilization. 



Table 1. Acceptance of E-learning Utilization 



Categorization 


Questions 


Code 




1, 1 would find e-learning useful in my job. 


PEI 


Performance 


2, Using e-learning enables me to accomplish tasks more quickly. 


PE2 


expectancy 


3, Using e-learning increases my productivity. 


PE3 




4, If 1 use e-learning, I will increase my chances of getting a raise. 


PE4 




5, My interaction with e-learning would be clear and understandable. 


EE1 


Effort 


6, It would be easy for me to become skillful at using e-learning. 


EE2 


expectancy 


7, 1 would find e-learning easy to use. 


EE3 




8, Learning to operate e-learning is easy for me. 


EE4 




9, Using e-learning is a good idea. 


ATI 


Attitude towards 


10, e-learning makes work more interesting. 


AT2 


E-learning 


11, Working with e-leaming is fun. 


AT 3 




12, 1 like working with e-leaming. 


AT4 




13,1 have the resources necessary to use e-learning. 


FC1 


Facilitating 

Conditions 


14, 1 have the knowledge necessary to use e-learning. 


FC2 


15, E-learning is not compatible with other systems I use. 


FC3 


16, A specific person (or group) is available for assistance with e-learning 
difficulties. 


FC4 




1 7, People who influence my behavior think that I should use e-learning. 


SI1 




18, People who are important to me think that I should use e-learning. 


SI2 


Social influence 


19, The senior management of this business has been helpful in the use of 
e-learning. 


SI3 




20, In general, the organization has supported the use of e-learning. 


S14 




21,1 feel apprehensive about using e-learning. 


AX1 


Anxiety 


22, 1 hesitate to use e-learning because of making a mistake. 


AX2 




23, E-learning is somewhat intimidating to me. 


AX3 


Behavioral 
Intention to use 


24, 1 intend to take e-learning in the next 3 months. 


IU1 


25, 1 plan to take e-learning in the next 6 months. 


IU2 


26, 1 predict I would take e-learning in the next 6 months. 


IU2 



Data Analysis and Results 



Participants 

The data were collected for three weeks (September 27 to October 19) in 2010 from a food service 
company in South Korea. The online survey was distributed within the company by the HRD staff and 261 surveys 
were returned. Among them, 92 datasets were deleted due to incompleteness or error. Of the 170 completed surveys, 
52 were completed by males (30.6%), 107 (62.9%) by females and 1 1 (6.5%) were not answered. Most participants 
(92.9%) were in their twenties and thirties. Fifty-eight (34.1%) participants were employees, 71 (41.8%) managers, 
and 29 (17.1%) were store managers, while 12 (7.1%) participants did not indicate their position in this company. 
Nearly half of the participants (49.4%) resided in Seoul at the time the survey was taken. Almost half of the 
participants (49.4%) had never experienced e-learning and 82 (48.2%) participants had experienced e-learning in the 
workplace. See Table 2 for demographics of the respondents. 



270 





Table 2. Demographics of the respondents (N = 170) 



Demographics 




Number (%) 


Demographics 




Number (%) 


Gender 


Male 


52(30.6) 


E-Learning Experience 


Yes 


82(48.2) 




Female 


107(62.9) 




No 


84(49.4) 




Missing 


11(6.5) 




Missing 


4(2.4) 


Age 


<29 


118(69.4) 


Position 


Employee 


58(34.1) 




30-39 


40(23.5) 




Manager 


71(41.8) 




40-49 


1(0.6) 




Store Manager 


29(17.1) 




Missing 


11(6.5) 




Missing 


12(7.1) 


Work Experience 


<2 years 


54(31.8) 


Workplace Location 


Seoul 


84(49.4) 




2-5 years 


78(45.9) 




Gyonggi 


35(20.6) 




5-10 years 


24(14.1) 




Daejeon 


3(1.8) 




>10 years 


2(1.2) 




Busan 


13(7.6) 




Missing 


12(7.1) 




Chungcheong 


3(1.8) 










Gyengsang 


4(2.4) 










Jeolla 


16(9.4) 










Missing 


12(7.1) 



Data reduction and Instrument validation 

The data were examined with factor analysis to validate and reduce the data. A principal component 
analysis (PCA) (KMO =.906) was conducted. An initial factor extraction was and rotated according to the varimax 
method. The PCA extracted 5 components with eigenvalues than 1.00 and accounted for 74.7 % of the variance. 
Table 3 shows the remaining five factors (23 items), which are factor 1, factor 2, factor 3, factor 4, and factor 5 and 
their reliability. The internal consistencies of the five factors vary from 0.851 to 0.962 (Table 4), which is good 
because an instrument is generally considered reliable when they have an alpha of .80 or higher on a scale of 0 to 1 
(Rubin & Babbi, 2009). 



Table 3. Total Variance Explained 



Initial Eigenvalues 




Extraction Sums of Squared Loadings 


Component Total % of Variance Cumulative % 


Total % of Variance Cumulative % 


1 12.738 


48.993 


48.993 




12.738 


48.993 48.993 


2 2.595 


9.982 


58.974 




2.595 


9.982 58.974 


3 1.900 


7.308 


66.282 




1.900 


7.308 66.282 


4 1.107 


4.258 


70.540 




1.107 


4.258 70.540 


5 1.084 


4.170 


74.710 




1.084 


4.170 74.710 


Extraction Method: Principal Component Analysis 








Table 4. Convergent validity 












Constructs/factors 


Indicators 




Loadings(>.60) 


Reliability(Cronbach's Alpha) 


Factor 1 


PE3 




.822 




.946 


Performance Expectancy 


AT2 




.784 






Attitude towards e-learning 


PE2 




.776 








AT3 




.761 








EE1 




.751 








PE4 




.745 








AT4 




.719 








PEI 




.702 








ATI 




.611 






Factor 2 


EE4 




.819 




.908 


Effort Expectancy 


EE3 




.796 






Facilitating Condition 


FC2 




.744 








EE2 




.698 







271 







FC1 


.661 




Factor 3 


IU2 


.902 


.962 


Intention to use 


IU3 


.891 






IU1 


.844 




Factor 4 


AX3 


-.755 


.893 


Anxiety 


AX1 


-.729 






AX2 


-.713 




Factor 5 


SI1 


.661 


.851 


Social Influence 


SI2 


.622 






SI3 


.601 





Descriptive Statistics 

Research question one: what are employees’ acceptance levels towards e-learning in the workplace? 

Table 5 shows the descriptive statistics of the acceptance of employees towards e-learning. Factor 3 (Intention to use 
e-learning) (Mean = 4.66, SD =1.07), factor 1 (Performance expectancy and attitude towards e-learning) (Mean 
= .454, SD = .87) and factor 2 (Effort expectancy and facilitating condition) showed relatively high mean scores 
compared to factor 4 (Anxiety) (Mean = .342, SD = 1.04). 



Table 5. Descriptive statistics of the acceptance levels towards e-learning (7 Likert scale) 



Factor 


Minimum 


Maximum 


Mean 


Std. Deviation 


Factor 1 PE/AT 


1.00 


7.00 


4.54 


.87 


Factor 2 EE/FC 


1.00 


7.00 


4.53 


.94 


Factor 3 IU 


1.00 


7.00 


4.66 


1.07 


Factor 4 AX 


1.00 


7.00 


3.42 


1.04 


Factor 5 SI 


1.00 


7.00 


4.45 


.965 


Valid N (listwise)N =170 



Inferential Statistics 

Research question two: is there a significant difference in the acceptance levels of employees towards 
e-learning based on gender, age, work experience, position, and E-learning experience? 

Table 6 shows that there are statistically significant differences in the acceptance levels of employees based on 
gender. Males have high mean scores of factor l(t = 3.081, df = 157, p<.002*), factor 2(t=2.5 12, df=108.418, 
p<.013 *), factor 3(t = 3.491, df =157, p< .01**), and factor 5(t = 3.803, df^92.529, p<.01**). In addition, males 
have lower anxiety levels towards e-learning than females (t= -3.591, df = 90,217, p<.01**). See Table 6 for detail 
information. 



Table 6. Employees’ E-learning Acceptance based on gender (T-test) 



Gender 




N 


Mean 


Std. 

Deviation 


t 


df 


Sig (2-tailed) 


Factor 1 PE/AT 


Males 


52 


4.8098 


.88762 


3.081 


157 


.002* 




Females 


107 


4.3769 


.80251 








Factor2 EE/FC 


Males 


52 


4.7885 


.86538 


2.512 


108.418 


.013* 




Females 


107 


4.4112 


.93439 








Factor3 IU 


Males 


52 


5.0833 


1.11998 


3.491 


157 


.001** 




Females 


107 


4.4579 


1.02946 








Factor4 AX 


Males 


52 


2.9808 


1.08949 


-3.591 


90.217 


.001** 




Females 


107 


3.6168 


.95584 








Factor5 SI 


Males 


52 


4.8718 


.98387 


3.803 


92.529 


.000** 




Females 


107 


4.2586 


.88938 









*P<0.5, * */?<0 . 0 1 
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Table 7 shows the mean differences between employees’ age and their acceptance towards e-learning. 

There are significant differences of factor 2(EE/FC), factor 3(IU), and factor 4(AX). Those in their thirties felt that 
they were comfortable and less anxious when they were using e-leaming in the workplace compared to employees in 
their twenties. In addition, participants in their thirties have stronger intention to use e-learning in the workplace 
compared to employees in their twenties. However, there are no significant differences in factor 1(PE/AT) and 
factor 5 (SI) towards e-learning according to age. See Table 7 for detailed information. 



Table 7. Age and the acceptance of employees towards e-learning (ANOVA) 



Age 




N 


Mean 


Std. Deviation 


F 


df 


Sig (2-tailed) 


Factor 1 PE/ AT 


>29 


118 


4.47 


.82 


.898 


2, 156 


.409 




30-39 


40 


4.68 


.94 










40-49 


1 


4.44 










Factor2 EE/FC 


>29 


118 


4.41 


.86 


4.358 


2, 156 


.014* 




30-39 


40 


4.90 


1.02 










40-49 


1 


4.20 










Factor3 IU 


>29 


118 


4.53 


1.01 


3.539 


2, 156 


.031* 




30-39 


40 


5.05 


1.27 










40-49 


1 


5.00 










Factor4 AX 


>29 


118 


3.55 


.96 


4.575 


2, 156 


.012* 




30-39 


40 


2.99 


1.18 










40-49 


1 


4.00 










Factor 5 SI 


>29 


118 


4.40 


.912 


1.072 


2, 156 


.345 




30-39 


40 


4.63 


1.10 










40-49 


1 


5.00 











*P<0.5, **p<0.01 



In terms of work experience, there are no significant differences in the acceptance of employees towards 
e-learning except anxiety levels. Participants who have longer work experience felt that they were more comfortable 
when they use e-learning with low anxiety levels (F = 2.812, df = 3, 154, p<.041 *). See Table 8. 



Table 8. Work experience and the acceptance of employees towards e-learning (ANOVA) 



Work experience 


N 


Mean 


Std. Deviation 


F 


df 


Sig( 2-tailed) 


Factor 1 


Within 2 


54 


4.45 


.72372 


.203 


3, 154 


.894 




2-5 


78 


4.54 


.95117 










6-10 


24 


4.60 


.84608 










11-15 


2 


4.67 


.62854 








Factor 2 


Within 2 


54 


4.36 


.78732 


1.787 


3, 154 


.152 




2-5 


78 


4.56 


.98263 










6-10 


24 


4.83 


.97920 










11-15 


2 


5.10 


1.27279 








Factor 3 


Within 2 


54 


4.52 


.86051 


1.114 


3, 154 


.345 




2-5 


78 


4.66 


1.19278 










6-10 


24 


4.96 


1.24843 










11-15 


2 


5.33 


.94281 








Factor 4 


Within 2 


54 


3.54 


.90286 


2.812 


3, 154 


.041* 




2-5 


78 


3.47 


1.05332 










6-10 


24 


3.01 


1.20978 










11-15 


2 


2.00 


.00000 








Factor 5 


Within 2 


54 


4.34 


.79435 


.793 


3, 154 


.500 




2-5 


78 


4.50 


1.04826 










6-10 


24 


4.65 


1.01904 










11-15 


2 


4.00 


1.41421 









*P<0.5, **p<0.01 
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An employee holds the lowest position and a manager holds a higher position than an employee. Franchise 
store managers are usually selected among managers. In terms of position, there are statistically significant 
differences among positions of factor 2(F = 1.355, df=2, 155, p<.002**), factor 3(F=3.902, df = 2, 155, p<.022*), 
and factor 4 (F = 5.435, df = 2, 155, p<.005**). In particular, there are statistically significant differences between 
employees and store managers in anxiety levels. Interestingly, employees have higher anxiety levels than store 
managers. Store managers felt that they were more comfortable using e-learning. Another significant difference in 
the intention to use e-learning was between managers and store managers. Store managers have stronger intention to 
use e-learning compared to managers. See Table 9 for detailed information. 



Table 9. Position and the acceptance of employees towards e-learning (ANOVA) 



Position 




N 


Mean 


Std. Deviation 


F 


df 


Sig( 2-tailed) 


Factor 1 PE/AT 


Employee 


58 


4.43 


.72973 


1.355 


2,155 


.261 




Manager 


71 


4.48 


.95363 










Store manager 


29 


4.74 


.77566 








Factor 2 EE/FC 


Employee 


58 


4.27 


.75616 


6.471 


2,155 


.002** 




Manager 


71 


4.55 


.96033 










Store manager 


29 


4.99 


.96582 








Factor 3 IU 


Employee 


58 


4.55 


.81303 


3.902 


2,155 


.022* 




Manager 


71 


4.54 


1.22860 










Store manager 


29 


5.16 


1.14303 








Factor 4 AX 


Employee 


58 


3.62 


.93752 


5.435 


2,155 


.005** 




Manager 


71 


3.47 


1.02288 










Store manager 


29 


2.87 


1.13184 








Factor 5 SI 


Employee 


58 


4.34 


.90373 


.885 


2,155 


.415 




Manager 


71 


4.49 


.97400 










Store manager 


29 


4.62 


1.06430 









*P<0.5, **p<0.01 



Table 10 shows that there is no significant difference in the acceptance of employees towards e-learning 
based on their workplace location. To determine whether there were differences in location variables, the analysis of 
variance (ANOVA) was performed. As a result, there was no relationship between the acceptance of employees 
towards e-leaming and locations. 



Table 10. Work location and the acceptance of employees towards e-learning (ANOVA) 



Position 




N 


Mean 


Std. Deviation 


F 


df 


Sig( 2-tailed) 


Factor 1 PE/AT 


Seoul 


84 


4.5317 


.88437 


.690 


6, 151 


.658 




Gyonggi 


35 


4.6413 


.95812 










Daejeon 


3 


3.8889 


1.05993 










Busan 


13 


4.5128 


.66441 










Chungcheong 


3 


3.8889 


.19245 










Gyengsang 


4 


4.3333 


.48005 










Jeolla 


16 


4.5000 


.69448 








Factor 2 EE/FC 


Seoul 


84 


4.6357 


1.00320 


1.094 


6, 151 


.368 




Gyonggi 


35 


4.5886 


.90741 










Daejeon 


3 


4.6000 


.87178 










Busan 


13 


4.3231 


.81461 










Chungcheong 


3 


3.8667 


.23094 










Gyengsang 


4 


3.7500 


.57446 










Jeolla 


16 


4.4000 


.71554 








Factor 3 IU 


Seoul 


84 


4.7381 


1.14430 


.481 


6, 151 


.822 




Gyonggi 


35 


4.7238 


1.01455 










Daejeon 


3 


3.8889 


.50918 










Busan 


13 


4.5897 


1.08997 










Chungcheong 


3 


4.3333 


.57735 










Gyengsang 


4 


4.3333 


.66667 
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Jeolla 


16 


4.5208 


1.30508 








Factor 4 AX 


Seoul 


84 


3.3810 


1.09293 


.280 


6, 151 


.945 




Gyonggi 


35 


3.2952 


1.12247 










Daejeon 


3 


3.7778 


.50918 










Busan 


13 


3.5641 


1.06619 










Chungcheong 


3 


3.3333 


.57735 










Gyengsang 


4 


3.3333 


.47140 










Jeolla 


16 


3.6042 


.88793 








Factor 5 SI 


Seoul 


84 


4.4841 


1.03668 


.611 


6, 151 


.722 




Gyonggi 


35 


4.5143 


.96793 










Daejeon 


3 


3.8889 


.19245 










Busan 


13 


4.6154 


.85901 










Chungcheong 


3 


3.6667 


.57735 










Gyengsang 


4 


4.3333 


.47140 










Jeolla 


16 


4.3958 


.87109 









*P<0.5, **p<0.01 



Table 1 1 shows a significant difference in the acceptance levels of employees towards e-learning based on 
e-learning experience. Participants who have experienced e-learning showed high factor 2 (EE/FC) (t = 4.97, df = 
157.96,/? < .01**), factor 3 (IU) (t = 4.95, df = 156.56, p < 0.01**), and low scores of factor 4 (AX) compared to 
participants who have never experienced e-learning in the workplace, which are statistically significant differences. 



Table 11. Employees’ E-learning Acceptance based on e-learning experience (/‘-test) 



E-experience 




N 


Mean 


Std. Deviation 


t 


df 


Sig (2-tailed) 


Factor 1 PE/ AT 


Experience 


82 


4.67 


.93 


1.881 


159.725 


.062 




No experience 


84 


4.42 


.81 








Factor2 EE/FC 


Experience 


82 


4.87 


.95 


4.967 


157.960 


.000** 




No experience 


84 


4.19 


.80 








Factor3 IU 


Experience 


82 


5.06 


1.11 


4.952 


156.564 


.000** 




No experience 


84 


4.28 


.91 








Factor4 AX 


Experience 


82 


2.97 


1.07 


-5.684 


153.759 


.000** 




No experience 


84 


3.82 


.84 








Factor5 SI 


Experience 


82 


4.60 


.98 


1.845 


164 


.067 




No experience 


84 


4.33 


.92 









*P<0.5, **p<0.01 



Research questions three: which factors affect employees’ intention to use e-learning in the workplace? 

The acceptance levels as independent variables affect the employees’ intention to use e-learning (R 2 = .357). 
In particular, factor 1 (PE/AT) and factor 4 (AX) influence employees’ intention to use e-learning in the workplace. 
See Table 12. 



Table 12. Factors affecting the intention to use e-learning in the workplace 



Model 


Unstandardized 
Coefficients 
B Std. Error 


Standardized 

Coefficients 

Beta 


t 


Sig 


df 


F R 2 


(Constant) 








5.365 


.000 


4, 165 


22.854 .357 


Factor 1 (PE/AT) 


3.706 


.691 


.237 


2.210 


.028* 






Factor 2 (EE/FC) 


.291 


.132 


.012 


.134 


.893 






Factor 4 (AX) 


.014 


.106 


-.330 


-4.034 


.000** 






Factor 5 (SI) 


-.339 


.084 


.146 


1.570 


.118 







*P<0.5, **/K0.01 

a. Predictors: (Constant), Factor 5(SI), Factor 4 (AX), Factor 2 (EE/FC), Factor 1 (PE/AT) 

b. Dependent Variable: Factor 3(the intention to use e-learning) 



275 





Conclusion and Discussion 



This study sought to investigate the acceptance levels of employees towards e-learning in the workplace of 
South Korea. It is encouraging to learn that most employees possess the intention to use e-learning in the workplace 
for job-related learning. Comparison on the acceptance levels of employees based on variables further generated 
numerous interesting findings. In particular, participants who have never experienced e-learning felt high anxiety 
levels when they use e-learning. Employees who are younger females, holding low positions in their workplace, or 
having less work experience felt a high anxiety level when using e-learning. In contrast, participants who have 
experienced e-learning, possess more work experience, and occupy high positions, felt confident in using e-learning. 
However, employees’ workplace location was not a factor that influences their acceptance levels towards e-learning. 
Among five factors, factor 1(PE/AT) and factor 4(AX) had the highest influence on employees’ intention to use 
e-learning. Factor 2 (EE/FC), and factor 5(SI) did not affect employees’ intention to use e-learning. 

This study identified a gender difference in the acceptance levels towards e-learning, which should be 
considered a critical factor in understanding how employees might accept e-learning in the workplace. Male 
participants rated high on factor 1(PE/AT), factor 2(EE/FC), factor 3(IU), and factor 5(SI), but low on factor 4 (AX). 
These findings supported the previous research that female users had higher Internet anxiety than male users 
(Chou, 2003). In terms of age, employees in their twenties surprisingly reported a higher level of anxiety towards 
e-learning than employees in their thirties. Brown et al., (2010) explained that age and attitudes towards using 
computers were moderately correlated. User attitudes, however, may become more positive with age. This finding 
could also be contributed by the utilitarian purpose of using e-learning systems for job-related learning. 

With regard to prior experiences, Mungania (2003) suggested that e-learning experience is not a significant 
predictor of e-learning barriers. However, the findings of this study indicated otherwise. Employees’ previous 
experience in using e-leaming contributed to a significant difference in factor 2(EE/FC), a high intention to use 
e-learning (factor 3:IU), and decreased anxiety (factor 4:AX). 

Limitations and Future Studies 

This study was conducted in only one company. Findings are based on the perceptions of employees who 
voluntarily chose to respond to the questionnaire. Therefore, participants may not be representative of all the 
employees in the company. In addition, the data were collected through a self-reporting mechanism, as opposed to 
direct observation. Given the small sample size it is inadequate to generalize our findings within this limited scope. 
The preliminary results, however, offer potential research directions to continuously improve the study in the future. 
Employees had an overall high performance expectancy, positive attitude toward using e-learning, and high 
intention to use e-learning in the workplace. In addition, factor 1(PE/AT) and factor 4(AX) influence employees’ 
intention to use e-learning in the workplace. However, they seemed to have different perspectives towards e-learning. 
Generally, gender, age, position, work experience, or e-learning experience makes a difference on acceptance levels 
towards e-learning in the workplace. It was found that there seems to be an intrinsic belief in the usefulness of 
e-learning for learning. Two factors provide a clue regarding how organizations seek to facilitate the utilization of 
e-learning in the workplace to support employees’ learning. To promote the utilization of e-learning in the 
workplace, these factors for developing strategies will be considered by HRD staff. Taking this idea further, we 
would like to examine what factors affect employees’ usefulness and reduced anxiety of e-learning in the workplace. 
Future research needs to take both factors into consideration. Employees’ acceptance level of e-learning will be 
examined before and after utilization of the tool for a learning activity. 

References 

American Society for Training and Development (2010). State of industry ; report: Executive summary >. Alexandria, 
VA: ASTD. 

Barron, A. E. (2003). Course management systems and online teaching. Journal of Computing in Higher Education, 
15(1), 128-42. 

Bates, T. (2005). Technology’, e-learning and distance education. London; New York: Routledge. 

Bates, T. (2010). The strategic management of e-learning in universities and colleges. 

http://www.editlib.org/?paper_id=35977&fuseaction=Reader.ViewPresentation&zoomed=true 
Biech, E. (2008). ASTD handbook for workplace learning professionals. 

Brown, T. et al., (2010) Predictors of attitudes to e-learning of astralian health science students. Journal of Applied 
Research in Higher Education, 2(1), 60. 



276 




Byun, S., & Lee, S. (2007). Evaluation e-learning institutions, Korean Research Institution for Vocational Education 
and Training. 

Chou, C. (2003). Incidences and correlates of internet anxiety among high school teachers in Taiwan. Computers in 
Human Behavior, 19(6), 731. 

Gunasekaran, A., McNeil, R. D., & Shaul, D. (2002). E-learning: Research and applications. Industrial and 
Commercial Training, 34(2), 44-53. 

Ingram, H., Biermann, K., Cannon, J., Neil, J., & Waddle, C. (2000). Internalizing action learning: a company 
perspective. Establishing critical success factors for action learning courses. International Journal of 
Contemporary Hospitality Management, 12(2), 107-113. 

Jia, H., Wang, M., Ran, W., Yang, S. J. H., Liao, J., & Chiu, D. K. W. (2011). Design of a performance-oriented 
workplace e-learning system using ontology. Expert Systems with Applications, 38 (4), 3374 - 3382. 

Johnson, R. D., Hornik, S., & Salas, E. (2008). An empirical examination of factors contributing to the creation of 
successful e-learning environments. International Journal of Human-Computer Studies, 66(5), 356-369. 

Keller, C., & Cernerud, L. (2002). Student’s Perceptions of e-Learning in University Education. Journal of 
Educational Media 27(1-2), 55-67. 

Keil, M. (1995). Pulling the plug: Software Project Management and the problem of project escalation. MIS 
Quarterly, 19(4), 421-447. 

Lee, S., Byun, S., Kwon, S., & Kwak, D. (2008). Developing Mid and long terms plan for e-learning, Korean 
Research Institution of Vocational Training and Education. 

Lee, B., Yoon, J., & Lee, I. (2009). Learners’ acceptance of e-learning in South Korea: Theories and results. 
Computers & Education, 53(4), 1320-1329. 

Liaw, S. S., Huang, H. M., & Chen, G. D. (2007). Surveying instructor and learner attitudes towards e-learning. 
Computers and Education, 49(4), 1066-1080. 

Link, T. M., & Marz, R. (2006). Computer literacy and attitudes towards e-learning among first year medical 
students. BMC Medical Education, 6, 34. 

Masrom, M. (2007). Technology Acceptance Model and E-learning. 12th International Conference on Education, 
Sultan Hassanal Bolkiah Institute of Education Universiti Brunei Darussalam. May 21-24. 

Masters, K., & Ellaway, R. (2008). AMEE Guide 32: e-Leaming in medical education Part 2: Technology, 
management and design. Med Teach 30, 474-489. 

Misko, J., Choi, J., Hong, S.Y., & Lee, I. S. (2005). E-learning in Australia and Korea: Learning from practice. 

Korea Research Institute for Vocational Education & Training and National Centre for Vocational 
Education Research, http://www.ncver.edu.au/research/core/cp0306.pdf . 

Mungania, P. (2003), The Seven E-learning Barriers Facing Employees, University of Louisville. 

Oshlyansky, L., Cairns, P. & Thimbleby, H. (2007). Validating the unified theory of acceptance and use of 
technology (UTAUT) tool cross- culturally. A Web document. Retrieved on 15 December 2007 at: 
http://www/users. cs. york. ac. uk/~ pcairns/papers/oshlyansky-Hcl07. Pfg 

Roca, J. C., & Gagne, M. (2008). Understanding e-learning continuance intention in the workplace: A 
self-determination theory perspective. Computers in Human Behavior, 24(4), 1585-1604. 

Rosenberg, M. J. (2001). E-learning: Strategies for delivering knowledge in the digital age. New York: 

McGraw-Hill. 

Rosset, A., & Marshall, J. (2010). E-learning: What’s old is new again. Training and Development, 1, 34 - 38. 

Rubin, A. & Babbie, E. (2009). Research Methods for Social Work (7th ed.). Pacific Grove, CA: Brooks Cole. 

Selim, H. M. (2003). An empirical investigation of student acceptance of course websites. Computers and Education, 
40(4), 343-360. 

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: 
Towards a unified view. MIS Quarterly, 27(3), 425-78. 



277 



The analysis of a learner’s interactions with messages on online 

social network services 

Yoon Hee, Kim 
Yonsei University 
50 Yonsei-ro, Seodaemun-gu 
Seoul 120-749, Korea 
vooon38@hotmail.com 



Abstract : This paper is a report on the findings of a learner’s interactions with messages on online social network 
services. The study was conducted on Twitter, one of the most popular social network services. Findings indicate 
that despite the limitations of space, various interactions among users were shown on social network services. 
Second, people could learn through interactions with other people in social network services. Third, there was much 
exchanging of information in following relationships; in contrast, more social exchanges 
and reflections appeared in non-following relations. 

Keywords : Content analysis, Online social network service, Learner’s interaction 

Introduction 

Unlike the majority of school education, adult learning is accomplished in various ways. Learning occurs in 
a real situation, and the process of learning depends on one's self-directed learning ability. The most important parts 
of adult learning are the processes of sharing and creating knowledge. In the process of learning, the sharing and 
creating of a learner's experience and knowledge through interaction are the most important processes of learning 
(Merriam & Brockett, 2007). 

An interaction between learners could take place when the learning process is active and easily accessible. 
Active learning contains plenty of interactions requiring learning spaces where learners could communicate. With 
the increasing usage of smart-phones, people can communicate in real-time and create lots of knowledge reflects 
their experience. With the creation of social network services, people began to talk in short messages to 
communicate, and share information with each other, which provides a new means by which creating and learning 
activities(Lewis et al. , 2010). Active communication between people who are not acquainted is the biggest 
characteristic of online social network services. This fits for social network theory, especially regarding “weak ties”, 
when there is little direct social exchange between people, they have diverse knowledge or thoughts due to different 
backgrounds and experiences. Even with weak ties, people who have common interest could interact easily in social 
network services, and the social network theory of information acquisition and learning shows sometimes weak ties 
could be more useful in sharing different thoughts and knowledge. People in close relationships have similar 
information and experiences are connected with “strong ties” through frequent social interaction. With strong ties, 
the creation of new knowledge through people from different environments is difficult compared to in the context of 
a weak connection in which various types of interactions can be expected(Burt, 2005). 

The use of social network services is expected to increase continually, so it is important to pay attention to 
interactions in social network services between learner and the learning process. This study focuses on the use of 
online social network services in social learning and uses the social network analysis perspective to evaluate how 
interaction affects learning by analyzing the current most popular online social network service, Twitter, where 
people actively interact through short messages. 

People exchange their thought and experience with others in Twitter by following relationship which 
means one person connect to another person in online space. A follower can see the other person’s message in his or 
her Twitter account by following others. The Twitter user analyzed in this study is a Korean novelist with a wide 
following of thousands of followers who writes novels through reflective interaction on his Twitter account. To 
analyze the messages of the novelist’s Twitter, all messages on his Twitter were gathered June to September, 2010, 
820 messages were collected. To study a learner’s interaction based on the strength of a relationship, a content 
analysis of the 820 messages was conducted. 
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The research questions are as follows: 

1) Does a learner’s interaction occur frequently in online social network service “Twitter ”? 

2) If an interaction of the learner’s occurs in Twitter, how does it appear differently in inter-relationship 
by strength of connection(strong ties, weak ties)? 

Background 



Learning With Social Media 

With the development of mobile media, interaction in online learning process is increasingly gaining 
importance. Twitter became one of the most popular online social network service with the usage of a smart-phone. 
As many people use Twitter with their individual mobile media, the opportunities for communication and sharing of 
knowledge increases(Hughes & Palen, 2009). 

At the same time, expend of social media makes a learner’s reflection and creation of knowledge faster. By 
using social media, patterns of a learner’s participation in learning changes rapidly. A learner can participate in the 
learning process by using online social media while on the move. It develops learners from being simply passive 
participants to being active participants who makes meaning with other learners. Social media is a place of specific 
contexts where many user makes meaning(Lewis, Pea, & Rosen, 2010). 

In many studies, researchers try to use social media in an educational way. So far, there are few studies that 
deal with a social media view of education theory. Instead of an academic view, many studies explain social media 
in a technical way, which ends description for method of using social network services or technical infrastructure for 
specific on-line social network services like Twitter or Facebook. Some studies examine micro-blogging services 
with a focus on using it as a learning tool. Few studies focus on the interaction between leamers(Ebner & Schiefner, 
2008; Ullrich et al, 2008). Most studies of micro-blogging services are analyzed technically or only by case 
study(Java et al, 2007). 

Social Network and Learning 

Social network theory, based on the human relations perspective, tries to explain human activity and the 
effect of social structure. According to the social network theory, the networks of each person’s interactions are 
produced and maintained by human activity, also the whole structure of network which produced by relation of 
individual affects their behavior(Kim, 2003). The degree of intimacy of the relationship, based on the relationship, 
can be divided into "strong ties" and "weak ties." In strong ties, there is a high frequency of contact, on weak ties 
have fewer contact than strong ties. Frequently, weak ties are more effective than strong ties because people could 
get new information from others met by accident, compared to people already know each other who have common 
interest and same information(Granovetter, 1994). 

Social network in learning is not a new concept. In particular, plenty of research has been conducted in a 
cooperate learning situation to investigate the relationship between a learner's interaction and social network(Cho, 
2009). By using social media, the boundary of social network is expanded. Also, a learner's knowledge and 
experiences increase rapidly. Existing popular social media, like blogs and communities, have strong relationships 
between users who have common interests or intimate relationships with each other. In contrast, the relationship 
based on a social network service is connected directly or indirectly one person to the other who has more 
information or knowledge than other users. Even if there was no direct exchange, users could form man-to-man 
connections by weak connections. In meaningful learning, weak connections could be more powerful than strong 
connections, and micro-blogging services such as Twitter have great potential in learning, which is worth 
investigating. 

Social networks in micro-blogging services shows new trends in a learner’s relationships, which consist of 
the interactions of learners. From the past, in communities and blogs, intimate relationships between people are 
shown or people tied with common interests have strong associations. In contrast, a user’s relationship on micro- 
blogging services is formed easily, even if there was no exchange of a man-to-man offline connection, people could 
exchange their ideas and emotions and be classified as "weak ties." In meaningful learning, the weak connection is 
sometimes more effective than a strong connection. It is important to pay attention to the learning that takes place on 
Twitter through the interactive communication between learners. 
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Analysis of Online Interactions and Messages 

A learner's interactions are such a major component in learning on internet space, to study the interaction of 
an online learner, content analysis of a learner's messages was conducted. To analyze online messages, 

Henri’s(1992) or Gunawardena’s(1997) model is mainly used. Gunawardena’s(1997) content analysis model, which 
is based on constructivism, contains steps to obtain new knowledge: first, the sharing/comparing of information 
second, the discovery and exploration of dissonance or inconsistency among ideas, concepts, or statements third, a 
negotiation of meaning/co-construction of knowledge and fourth, the testing and modification of a proposed 
synthesis or co-construction. Each stage contributes to learning and the creation of knowledge. Another model by 
Henri(1992) consists of five categories. Henri distinguished online messages into the participative, social, interactive, 
cognitive, and meta-cognitive messages. Until now, these two online message analysis frameworks have been used 
generally, in spite of some shortcomings. The framework based on Henri's model has ambiguity between each 
categories and sometimes needs to divide more detail categories. Gunawardena's model explains steps of the 
learning process on internet space, but in the actual learning process, some steps are omitted or, in many cases, 
disappear altogether. Researchers have tried to overcome the imperfections of the existing models many of them 
redefined exist message analysis model in their studies (Yoon et ah, 2008; Choi et ah, 2010). 

This study focuses on the most popular online social network service Twitter in which people communicate 
via short messages containing only 140 characters. People can easily access Twitter by using their mobile phones, 
multiple messages are generated in a short time. To investigate the interaction of learners, Henri's model was 
considered more appropriate for this study. 



Methods 

Data for analysis was gathering from the Twitter account of a famous Korean novelist who likes to interact 
with people to reflect on the contents of a novel. Data was collected from June to September, 2010, 820 messages 
were collected. There were no messages using the Twitter longer service which make users could write more than 
140 characters. In this novelist’s Twitter, there was plenty of discourse with many people who use the Twitter 
service for interactive conversation, and sometimes debate, for social topics. The analysis of the messages of online 
social network services was conducted using Henri's(1992) classification criteria for online discourse, which 
consists of five categories. In this study cognitive and meta-cognitive message were analyzed together at the 
category of reflective messages, in short messages it’s difficult to distinguish cognitive and meta-cognitive messages. 
At the first coding, the 820 messages classified to four categories with the exception of meta-cognitive messages. In 
the second coding, using the framework for analysis, the messages divided into seven categories, which consist of 
the following: simple information, simple emotion, social, request information, provide information, and reflective 
messages, with the number of messages in each groups counted. An example of each category is shown in Table 1. 
Two researchers analyzed messages separately using the same criteria. Scores between two researchers showed 93% 
concordance. About 57 messages did not match each other, and the two researchers determined each message 
category by discussion. 



Table 1 Message categories of online social network services 



Category 


Description 


Sub-category 


Examples 


Participative 

message 


Make learner 
to participate 


Derive 

participation 


"Prohibition of discipline is a violation of education law, but many 
people said that without discipline it’s difficult to maintain order of 
the education field. Does education law ensure the physical 
violation from teacher to students?” 


Simple 

fact 


"In my new house, there is no internet, no television, no cell- 
phone." 


Simple 

emotion 


"Wow! It’s so cool!” 


Social 

message 


Learner’s discourse 
doesn’t have direct 
relevance to learning 


Social 

interaction 


“Yes, it’s been a long time. You have a new Twitter account. How 
you do in’?" 


Interactive 

message 


Exchange message 
between learners 
(answer & question) 


Request 

information 


"I’m wondering if the source of Korean’s reading quantity is lower 
than any other country. Is there any evidence? Where does the 
statistics data come from?" 
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Provide 

information 


"Do you know the 00 i-Phone application? It makes use of the i- 
Pad as another monitor. If s so useful." 


Cognitive 

message 


Knowledge and skills 
related learning 


Reflection 


"I’m so sony that this Korean newspaper has a tendency to write 
articles without specific data. There are many immigrants and 
illiterate people in European countries, and so the reading quantity 
is lower than our prediction. For example, in France there are so 
many non-French users." 


Meta- 

cognitive 

message 


Learner’s reflection 
and perception 



Findings 

To study learning patterns in online social network services by the learner’s social exchange, an analysis of 
the number and content of messages was conducted. The number of messages in each category was arranged in the 
order of social messages, interactive messages, cognitive messages, participatory messages. Table 2 shows the 
results of the number of each category in the order of social messages(40.5%), interactive messages (24.2%), 
cognitive messages (20.8%), and participation messages(14.5%). Most of messages Twitter users create contains 
reflective, informative contents(64.3%). Meanwhile, answers to another Twitter user’s comments account for a large 
part of social messages(64.9%). Messages classified as "retweet" are message taken from another Twitter written by 
someone else. People usually retweet others when they want to keep or agree with the message. In this study, there 
were lots of messages related to provide information) 70.1%). 



Table 2 Type and frequency of messages on online social network services 



Category 


Sub-category 


Self 


Others 


Total 


Direct 


Answer 


Retweet 


Answer 




Derive participation 


9 


7 


2 


- 


18 (2.2%) 


Participative 

message 


Simple fact 


22 


7 


2 


2 


33 (4.0%) 


Simple emotion 


42 


24 


1 


1 


68 (8.3%) 


Social message 


Social exchange 


20 


310 


1 


1 


332 (40.5%) 


Interactive 


Request information 


4 


9 


- 


- 


13 (1.6%) 


message 


Provide information 


77 


58 


47 


3 


185 (22.6%) 


Cognitive message 


Reflection 


89 


63 


14 


5 


171 (20.8%) 


Total 


263 


478 


67 


12 


820 (100%) 



Twitter users left reflective messages using simple sentences. In the process of knowledge creation, due to 
the nature of the 140 character short message and the ability to use it anywhere in the world, people leave simple 
messages voluntarily and express themselves in a direct way. Therefore, there are fewer participation messages than 
the general online community. In contrast, social messages ask to greeting and regards are shown frequently. 

To investigate the differences of a learner’s interaction depending on the strength of the connection, a 
frequency analysis of messages each occurred in strong and weak connections was conducted. At first, 820 
messages were targeted for the analysis, but 263 messages which Twitter user wrote alone did not show any 
interaction with others. Therefore, this study analyzed 557 messages which showed connections to other people. 



Table 3 Type and frequency of messages on online social network services by strength of relationship 



Category 


Sub-category 


Strong connection 
(following) 


Weak connection 
(non-following) 


Total 


Answer 


Retweet 


Total 


Answer 


Retweet 


Total 






Derive participation 


1 


2 


3(1.4%) 


6 


- 


6(1.8%) 


9(1.6%) 


Participative 

message 


Simple fact 


3 


1 


4(1.8%) 


4 


3 


7(2.0%) 


11(2.0%) 


Simple emotion 


12 


1 


13(6.0%) 


12 


1 


13(3.8%) 


26(4.7%) 
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Social message 


Social exchange 


112 


1 


113(52.1%) 


198 


1 


199(58.5%) 


312(56.0%) 


Interactive 


Request information 


2 


- 


2(0.9%) 


7 


- 


7(2.1%) 


9(1.6%) 


message 


Provide information 


17 


36 


53(24.4%) 


41 


14 


55(16.2%) 


108(19.4%) 


Cognitive message 


Reflection 


17 


12 


29(13.4%) 


46 


7 


53(15.6%) 


82(14.7%) 


Total 


164 


53 


217(100%) 


314 


26 


340(100%) 


557(100%) 



In non-following relationships, the ratio of social (58.5%) and cognitive (15.6%) messages were higher 
than through following relationships in which social (52.1%) and cognitive (13.4%) messages were seen. When 
people had close relationships, they used message to exchange information (24.4%) rather than to say hello or have 
emotional discourse. Strong connections with other users showed a high ratio of information exchange messages 
over expressions of shared feelings and thoughts. There were especially many retweet messages Twitter user scrap 
others message to share information. In contrast, weak connections with other users showed that they communicated 
based on the messages they are interested in, Twitter user doing social interaction or supply information in the 
process of answering his or her non-followers mention. 

Conclusions 

This study applied Henri's classification for analyzing electronic discussions on online social network 
service “Twitter”. The result of this study revealed that interactions between social network service users occurred in 
various ways. People exchange information, feelings, and their thoughts on Twitter, which could create new 
meanings and knowledge. The Twitter service plays a role in the construction of knowledge in the context of each 
person who participated in electronic discussion. Although space was limited to 140 characters, lots of reflective 
messages, which play major roles in learning, appeared on Twitter. In addition, the creation of meaning and 
knowledge place often between people with weak ties rather than strong ties. 

From this study, suggestions for further research are proposed as follows. First, research on the message of 
online social network services needs more in-depth studies. In this study, the scope of analysis was one's Twitter 
account, which contains messages related to one person who was at the center of the electronic discussion. In 
addition, the analysis of other people's Twitter accounts who are related to the person in the center of discourse 
should be carried out simultaneously. Second, with the emergence of new media, a new framework for content 
analysis should be considered. Finally, this study includes an analysis of the contents of the messages focused on 
quantitative analysis. Reflections in messages were contained in the same category, even though they may have 
shown different levels of reflection. Through a qualitative analysis of the contents of the messages, we can reveal 
the detail process of learning on online social network services, something that is necessary to create new meaning 
and knowledge. 
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Abstract: This paper investigated whether distance learning faculty members have adopted best 
practices in their distance learning courses via a self-report survey. It provided insights regarding 
relationships between demographic characteristics and the perceived utilization of the evaluated 
best practices of course design and delivery in their distance learning courses. Such information is 
beneficial to academic staff when planning faculty support and development activities. 



Introduction 

The growth of distance learning programs is tremendous. A number of best practices have been created and 
employed by universities and educational agencies. Tables 1 and 2 provide an overview of these standards, 
benchmarks, or best practices. However, the literature to date has had little to say about whether distance learning 
faculty members have applied these best practices in their distance learning courses. To ensure the quality of a 
distance learning program, it is critical to examine that whether distance learning faculty have utilized best practices 
in their distance learning course design and delivery. 



Table 1 

Best practices of online teaching 


Bailey & Card (2009) 


• 


fostering relationship 




• 


engagement 




• 


timeliness 




• 


communication 




• 


organization 




• 


technology 




• 


flexibility 




• 


high expectations (p. 154) 


Murphy (2008) 


• 


support 




• 


collaboration and teamwork 




• 


flexibility 




• 


feedback 




• 


assessment 




• 


adult learning techniques (p. 105) 



Although technological tool is just a tool in teaching and learning, it plays a critical role for distance 
learning faculty members to implement the best practices evaluated in this study, for example without two-way 
interactive technologies it is impossible to promote student-student and student-faculty interaction in a timely 
manner. Caladine (2008) and Taylor (2000) identified six generations of distance education model and discussed the 
features of delivery technologies. The discussions of the characteristics of technological tools help us understand 
how different technologies can help faculty fully utilize these best practices in their distance learning course design 
and delivery. 
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Table 2 



Bench marks, standards, and best practices of distance education 



Higher Learning 
Commission (HLC, 
2007) 


• institutional context and commitment 

• curriculum and instruction 

• faculty support 

• student support 

• evaluation and assessment (p. 2) 


The Institute for 


• institutional support 


Higher Education 


• course development 


Policy (IHEP, 2000) 


• teaching/learning process 

• course structure 

• student support 

• faculty support 

• evaluation and assessment (pp.2-3) 


The Higher Education 


• faculty must retain academic control 


Program and Policy 


• faculty must be prepared to meet special requirements of teaching at a 


Council (HEPPC, 


distance 


2000) 


• course design should be shaped to the potential of the medium 

• students must fully understand course requirements and be prepared to 
succeed 

• close personal interaction must be maintained 

• class size should be set through normal faculty channels 

• course should cover all materials 

• experimentation with a broad variety of subjects should be encouraged 

• equivalent research opportunities must be provided 

• student assessment should be comparable 

• equivalent advisement opportunities must be offered 

• faculty should retain creative control over use and re-use of materials 

• full undergraduate degree programs should include same-time same- 
place coursework 

• evaluation of distance coursework should be undertaken at all levels (pp. 
7-15) 



Methodology 

After reviewing literature on best practices, the investigator found that there is a gap between the existing 
best practices and the best practices that distance learning faculty members used in their course design and delivery. 
Based on the literature (Bailey & Card, 2009; Murphy, 2008; Bransford, Garrison, Anderson & Archer, 2000; 

Brown, Cocking, Donovan, & Pellegrion, 2000; Chickering & Gamson, 1987; HEPPC, 2000; HLC, 2007; Holmberg, 
1995; IHEP, 2000; Merrill, 2002, and QM, 2006) 12 themes are identified as best practices of distance learning 
course design and delivery: (1) encourage contact between students and faculty, (2) encourage cooperation among 
students, (3) use active learning technique, (4) give prompt feedback, (5) emphasize time on task, (6) communicate 
high expectations, (7) respect diverse talents and ways of learning, (8) give students opportunities for reflection, (9) 
provide learner support and resources, (10) adapt course organization and design for online delivery, (11) pay 
attention to legal, ethical and academic dishonesty, and (12) use objective-driven design concepts. 

Distance learning faculty members at two higher education institutions, were surveyed to answer the 
following questions: (1) To what extent do distance learning faculty members perceive that they have utilized the 
best practices in their distance learning courses? (2) Are there relationships between faculty perceptions of utilizing 
each of the evaluated best practices in distance learning course design and delivery and demographic characteristics? 
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The two institutions are located in the West and the Mid West. Both of them offer online undergraduate and 
graduate programs and provide instructional design services to instructors through instructional designers. The 
survey was conducted via SurveyMonkey.com. Email messages were sent to 525 distance learning faculty members. 
In total, 152 of 525 distance learning faculty members completed the survey within twelve business days. The 
response rate increased to 29% after two follow-up emails. 

Rasch Model was employed to inspect the data. A one-way analysis of variance (ANOVA) was used to 
possible differences among faculty members’ perceptions of best practices, and choosing to work or not to work 
with instructional designers based on demographic characteristics. Responses to the open-ended questions were 
reviewed and analyzed to reveal themes and patterns. The demographics of the participants are illustrated in tables 3. 

Table 3 



Participant demographic information 



Gender 


Frequency 


Percentage 


Male 


69 


45.4 


Female 


80 


52.6 



Academic Rank 



Professor 


33 


21.7 


Associate professor 


45 


29.6 


Assistant professor 


23 


15.1 


Lecturer 


12 


7.9 


Instructor 


3 


2.0 


Adjunct faculty /Part time instructor 


29 


19.1 


Teaching assistant/GA 


5 


3.3 



Academic field 



Arts and Humanities 


27 


17.8 


Business 


32 


21.1 


Education 


28 


18.4 


Engineering 


3 


2.0 


Science 


11 


7.2 


Social Science 


25 


16.4 


Health Science 


23 


15.1 



Years of experience teaching distance learning courses 



More than 4 years 


84 


55.3 


3-4 years 


25 


16.4 


2-3 years 


17 


11.2 


1-2 years 


19 


12.5 


Less than 1 year 


5 


3.3 



Results 

Rasch statistics of reliability, separation, item fit, and person fit were analyzed to better understand faculty 
members’ perceived utilization of each of the evaluated best practices in distance learning course design and 
delivery. Redundant items were removed from the data analysis. The reliability for person was 0.73 with a 
separation of 1.63, indicating there might be a trend toward two groups. The reliability for items was 0.94 with a 
separation of 3.95, indicating that the items were roughly separated into three groups. 

To what extent do distance learning faculty members perceive that they have utilized the best practices in 
their distance learning courses? 

As illustrated in the Rasch person-item map in Figure 1 (see Appendix A) respondents were likely to agree 
to the item in regard to course design. The result suggested that most faculty members believed that they organized 
their course content in a logical format, as shown in Level 1 — Content-centric in Figure 1. The items placed in Level 
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3 — Learner-centric, as shown in Figure 1, in regard to learner support, reflective learning, time on task, respect 
diverse talents, accessibility issues, and deter academic dishonesty were comparatively difficult statements with 
which to agree, indicating these best practices were not frequently used in distance learning courses as reported by 
the participants in this study. 

Table 4 provides the mean and standard deviation for each best practice in distance learning course design 
and delivery evaluated in the study. 



Table 4 

Individual Mean Scores for Each Best Practice 



Item 

1 . Promote faculty-student interaction 

2. Promote student-student interaction 

3. Use active learning activity 

4. Provide constructive feedback 

5. Give sufficient time on task 

6. Assign challenging tasks 

7. Allow self-chosen topics 

8. Use reflective learning 

9. Provide learner support 

10. Use logical design 

11a. Pay attention to accessibility issue 
1 lb. Pay attention to dishonesty-policy 
1 lc. Use strategies to deter dishonesty 
lid. Pay attention to copyright issues 
12. Use objective-driven design 



N 


Mean 


Std. Deviation 


152 


3.58 


.558 


152 


3.30 


.789 


152 


3.50 


.681 


151 


3.50 


.599 


151 


3.71 


.456 


151 


3.44 


.596 


148 


3.05 


.871 


148 


3.01 


.685 


151 


2.87 


1.015 


151 


3.66 


.491 


151 


2.91 


.840 


151 


3.52 


.773 


149 


2.91 


1.013 


148 


3.39 


.656 


150 


3.46 


.652 



Qualitative analysis of the open-ended question 

The open-ended question in the first part of the survey solicited other best practices that faculty members 
used in their distance learning courses. Forty participants (26.3%) offered comments and suggestions. The responses 
to the open-ended question revealed other best practices that include the following: 

1 . Collaborate with peers for course improvement or refer to other courses developed by peers for ideas. 

2. Continuously improve distance learning courses based on students’ performances and feedback. 

3. Create opportunities for students in creating student-owned content so that students have a sense of 
ownership for the content created, such as assigning student leaders for weekly discussions and learning 
blogs. 

4. Create opportunities to give students an orientation of the course, such as using an orientation quiz on the 
course syllabus. 

5. Give students more opportunities in practice quizzes or learning activities so that they can self-assess their 
own learning. 

6. Create opportunities for students to increase their time-management skills — for example, encouraging 
students to create their own schedules according to the course schedule. 

One faculty stated the following: 

I require my students to schedule their work on the Course Calendar to ensure that they know 
when assignments are due. The rationale is that if they have to enter the dates (which they find on 
the drop box, test center, and discussion board) it will increase their involvement and make them 
proactive when completing course work. I remind them that the date and time I place on 
assignments is not the "start" date — it is the last possible day and hour to submit the work. By 
encouraging students to learn scheduling and planning skills that work for them, it is a win-win 
situation for all of us. 
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Table 5 

Individual Frequency and Percentage of Each Best Practice 



Item 




Scale 


Frequency 


Percent 


1 . 


Promote faculty-student interaction 


1 


0 


0.0 






2 


5 


3.3 






3 


54 


35.5 






4 


93 


61.2 






Total 


152 


100.0 


2. 


Promote student-student interaction 


1 


4 


2.6 






2 


19 


12.5 






3 


56 


36.8 






4 


73 


48.0 






Total 


152 


100.0 


3. 


Use active learning activity 


1 


3 


2.0 






2 


7 


4.6 






3 


53 


34.9 






4 


89 


58.6 






Total 


152 


100.0 


4. 


Provide constructive feedback 


1 


0 


0.0 






2 


8 


5.3 






3 


59 


38.8 






4 


84 


55.3 






Total 


151 (missing 1) 


99.3 


5. 


Give sufficient time on task 


1 


0 


0.0 






2 


0 


0.0 






3 


44 


28.9 






4 


107 


70.4 






Total 


151(missing 1) 


99.3 


6. 


Assign challenging tasks 


1 


1 


.7 






2 


5 


3.3 






3 


71 


46.7 






4 


74 


48.7 






Total 


151 (missing: 1) 


99.3 


7. 


Allow student choose their project topics 


1 


11 


7.2 






2 


19 


12.5 






3 


69 


45.4 






4 


49 


32.2 






Total 


148(missing: 4) 


97.4 


8. 


Use reflective learning activity 


1 


4 


2.6 






2 


22 


14.5 






3 


91 


59.9 






4 


31 


20.4 






Total 


148 


97.4 


9. 


Provide learner support 


1 


18 


11.8 






2 


34 


22.4 






3 


48 


31.6 






4 


51 


33.6 






Total 


151(missing 1) 


99.3 


10. 


Use logical design 


1 


0 


0.0 






2 


1 


.7 






3 


50 


32.9 






4 


100 


65.8 






Total 


151(missing 1) 


99.3 
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Table 6 

Individual Frequency and Percentage of Each Best Practice continued 



1 la. Pay attention to accessibility issue 



lib. Pay attention to academic dishonesty issue 



1 lc. Deter academic dishonesty 



lid. Pay attention to copyright issue 



12. Use objective-driven design 



1 


9 


5.9 


2 


33 


21.7 


3 


71 


46.7 


4 


38 


25.0 


Total 


151 (missing 1) 


99.3 


1 


5 


3.3 


2 


11 


7.2 


3 


35 


23.0 


4 


100 


65.8 


Total 


151 (missing 1) 


99.3 


1 


15 


9.9 


2 


38 


25.0 


3 


41 


27.0 


4 


55 


36.2 


Total 


149 (missing 3) 


98.0 


1 


2 


1.3 


2 


8 


5.3 


3 


68 


44.7 


4 


70 


46.1 


Total 


148 (missing 4) 


97.4 


1 


1 


.7 


2 


10 


6.6 


3 


58 


38.2 


4 


81 


53.3 


Total 


150 (missing 2) 


98.7 



The responses to the open-ended question also reinforced the best practices of faculty-student interaction 
and time on task. 

Frequent interaction has been shown to be a major moderating factor in academic success in 
distance education (along with time on task). I communicate with every student individually in 
multiple ways and multiple times through-out the course and take a survey of appropriate 
background knowledge (i.e., technological, distance learning and learning preferences) at the 
beginning of the course. I am also not afraid to modify the course scope and sequence or the 
timing in order to accommodate the learning needs of a given class. The time commitment 
required to teach a quality distance education class is much greater than that required to teach the 
same class f2f. In addition, DE classes require technological expertise as well as content 
knowledge — and all teaching requires pedagogical expertise that is almost never taught to higher 
education instructors. 

Furthermore, the responses to the open-end question reinforced the best practices of student-student 
interaction. 

Students provide feedback to classmates using the "feedback sandwich" method: start with a 
positive comment, then add a constructive criticism, and finish with another positive comment. 

Are there relationships between faculty perceptions of utilizing each of the evaluated best practices in 
distance learning course design and delivery and demographic characteristics? 

One-way analysis of variance (ANOVA) was used to determine whether differences existed in perceptions 
of utilizing each of the evaluated best practices across gender, academic rank, academic field, and years of teaching 
distance learning courses. Differences existed across gender regarding learner support (F=5.821, p=.017) and 
objective-driven design (F=3.944, p=. 049), academic rank regarding active learning activity (F=2.432, p=.029) and 
academic dishonesty policy (F=2.241, p=,043), and academic field regarding student-student interaction (F=2.799, 
p=.019), reflective learning (F=3.550, p=. 003), and accessibility issue(F=2.774, p=.014). However, no statistical 
difference was found between each of the evaluated best practices and years of experience teaching online. 
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The analysis results for gender were significant on item 9 (learner support) and item 12 (objective-driven 
design). The remaining thirteen items about best practices yielded no significance. The analysis results for Item 9 
(provide learner support by adding an FAQ page or a discussion topic on the discussion board) were significant: F = 
5.821, p =.017. Female respondents (M = 3.08, SD =.956) were likely to agree with this item, that is to include an 
FAQ page or a discussion topic for projects or general questions on the course site. Male respondents (M=2.68, 
SD=1.043) were unlikely to agree with this statement. 

The analysis results for Item 12 (use objective-driven design technique in designing learning activities) 
were significant: F= 3.94, p=.049, as shown in Table 10. Female respondents (M = 3.55, SD =.571) strongly agreed 
with this item, that is to use learning objectives to design learning activities. 

The analysis results for academic rank were significant on Item 3 (active learning activities) and item lib 
(academic dishonesty policy). The remaining 13 items about best practices yielded no significance. The analysis 
results for Item 3 (use active learning activities) were significant: F= 2.432, p= .029. Lecturers, assistant professors, 
instructors, and teaching assistants strongly agreed with this item, which is to use active learning activities such as 
discussions, case studies, and simulations to engage students in their distance learning courses. Professors and 
associate professors agreed with this statement. 

The analysis results for Item 1 lb (Reference to the university academic dishonesty policy in the course) 
were significant: F= 2.241, p = .43. Lecturers, adjunct faculty, and teaching assistants strongly agreed with this item, 
which is to reference the university academic dishonesty policy in the syllabus or on the course site. Instructors, 
associate professors, and professors agreed with the statement. 

The analysis results for academic field was significant on Item 2 (student-student interaction), Item 8 
(reflective learning), and Item 1 la (accessibility issue). The results for the remaining 12 items about best practices 
yielded no significance. The analysis results for Item 2 (promote student-student interaction) were significant: F = 
2.799, p= .013. Faculty members of engineering and sciences were unlikely to agree with this item, that is to say, 
they were not likely to use Internet communication tools to encourage student-student interaction in their distance 
learning courses. 

The analysis results for Item 8 (use reflective learning activities) were significant: F = 3.550, p = .003, as 
shown in Table 12. Faculty members of social science, sciences, and health science were unlikely to agree with this 
item, that is to say, they were not likely to give students opportunities to reflect on their own learning throughout the 
semester, especially in the middle and at the end of a semester. Faculty members in education, business, arts and 
humanities, and engineering were likely to give students opportunities to reflect on their own learning. 

The analysis results for Item 1 la (accessibility issues) were significant: F = 2.774, p= .014. Faculty 
members of engineering and sciences were unlikely to agree with this item, that is to say, they were not likely to 
work with instructional designers to make sure that students with special needs could access course materials. One 
possible reason for this might be that no students with disabilities enrolled in a distance learning course taught by 
faculty in these fields. Faculty members of health science and business were likely to work with instructional 
designers to make sure that students with special needs could access course materials. 



Conclusions 

The following conclusions were drawn based on the data analysis: 

To what extent do distance learning faculty members perceive that they have utilized the best practices in 
their distance learning courses? 

Faculty members perceived that they have used each of the 12 best practices evaluated in this study. The 
best practices evaluated in this study can be divided into three levels: 

• Level 1 Content-centric — most used best practices 

• Level 2 Instructor-centric — somewhat used best practices, and 

• Level 3 Learner-centric — least used practices. 

As shown in Figure 1, the most used best practice is designing course content in a logical format, the 
somewhat used best practices are promoting student-student interaction, referring to the university academic 
dishonesty policy, providing constructive feedback, using active learning activities, promoting faculty-student 
interaction, using objective-driven design, and assign challenging tasks to students; and the least used best practices 
are providing learner support by adding FAQ or discussion topics, allowing students to choose their own project 
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topics, giving students opportunities to reflect on their own learning, paying attention to accessibility issues, and 
using assessment strategies to deter academic dishonesty. 

Are there relationships between faculty perceptions of utilizing best practices in distance learning course 
design and delivery and demographic characteristics? 

Differences existed across gender, academic rank, and academic field regarding some of the best practices 
evaluated in this study. Statistical analysis of data related to this question yielded the following conclusions: 

1. Female respondents were more likely to do the following in their distance learning courses than their male 

counterparts: 

1.1. Provide learner support. Female faculty members tended to use an FAQ page or discussion topic to 
provide learner support more than their male counterparts. 

1.2. Use objective-driven design. Female faculty members tended to use learning objectives to design 
learning activities for their distance learning courses more than their male counterparts. 

2. Faculty members with different academic ranks tended to disagree on the following best practices: 

2.1. Use active learning activities. Assistant professors, lecturers, instructors, and teaching assistants 
perceived they used active learning activities in their distance learning courses more compared to 
professors and associate professors. 

2.2. Reference the university academic dishonesty policy. All lecturers referred to the university academic 
dishonesty policy in their course syllabi or on their course sites. Assistant professors, adjunct faculty, 
and teaching assistants perceived they referenced the university academic dishonesty policy more 
compared to faculty with other academic ranks. 

3. Faculty members in different academic fields tended to implement differently in the following best 

practices: 

3.1. Use Internet communication tools to promote student-student interaction. Engineering and science 
faculty tended not to use Internet tools to promote student-student interaction, while faculty members 
in other academic fields tended to use communication tools to promote student-student interaction. 

3.2. Use reflective learning activities. Social sciences, sciences, and health science faculty members 
tended not to give students opportunities to reflect on their own learning, while faculty members in 
other fields tended to give students opportunities to reflect on their own learning. 



Summary 

Faculty members reported that they have utilized most of the best practices identified in the literature that 
were evaluated in this study. Differences across gender, academic rank, and academic field in some of the evaluated 
best practices exist. Therefore, instructional designers need to strive to consistently promote best practices to 
distance learning faculty members. Institutions that offer distance learning courses would benefit from providing 
instructional design services to faculty members, especially services in implementing these best practices in distance 
learning course design and delivery, to ensure quality distance learning courses and programs. Institutions should 
encourage, if not require, distance learning faculty members to work with instructional designers in a community of 
practice in designing and delivering their courses. 

This study provided a big picture of faculty members’ perceived unitization of each of the evaluated best 
practices in distance learning course design and delivery. A study on the actual use of these best practices conducted 
by certified distance learning course evaluators, such as Quality Matters master reviewers or peer reviewers, might 
provide greater understanding of what was reported by distance learning faculty members, as there may be a gap 
between the actual use and perceived utilization. 
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Appendex A : Figure 1 Hierarchal map of faculty and each of the evaluated best practices 
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Abstract 

Based on the Community of Inquiry model (Garrison, Anderson, & Archer, 2000), this study has 
examined the nature and interactions of the teaching, cognitive, and social presences fostered between an instructor 
and students in an Education-based Social Network Site developed by a large Midwestern university, Mixable. 
Scaffolds were developed in order to support varied students’ cognitive, social and teaching presences essential to 
the promotion of both critical discourse and social interaction in an Education-based Social Network Site 
environment. The study results also inform e-learning designers and instructors on the relationships that exist 
between online teaching, cognitive, and social presences. 

Introduction 

Educators in higher education are eager to introduce Web 2.0 as a means both to integrate informational 
access and to better connect the learners themselves. Social Network Sites (SNSs), in particular, have been used for 
the successful sharing of ideas as well as for socially-shaped scholarly reflections. However, there is little 
understanding as of yet on how SNSs can be successfully designed and implemented in regards to education. 
Therefore, it is important to provide college students with effective educational experiences in order to create an 
educational network with common interests or visions permitting said students to engage in learning and to better 
share in their customized learning experiences. This paper offers an introduction to an Education-based Social 
Network Site, entitled Mixable, developed by a large Midwestern university and examines its effectiveness for 
sustaining academic relationships by exploring the nature and interactions of the teaching, cognitive, and social 
presence practices created by online instructors and students in Mixable. In particular, the authors have focused on 
the development of concrete scaffolding to guide online instructors and e-learning designers based on the 
Community of Inquiry (Col) model. The Col model has been approached as an effective and efficient measurement 
of learning within online learning platforms (Akyol & Garrison, 2008; Burgess et al., 2010; Shea & Bidjerano, 
2009). It has also been used as an analysis framework for varied spectrum of social network content to understand 
the implications for online learning practices (Shea et ah, 2010). In this paper, the authors are examining Mixable 
through the lens of the Community of Inquiry (Col) framework in order to determine effective strategies of Social 
Network Sites (SNSs). 



Literature Review 



Social Network Sites (SNSs) 

As a dominant portion of the Web 2.0 era, social network sites (SNSs) such as MySpace and Facebook 
stand in the spotlight of numerous web users in their daily life. While most sites are used to maintain existing off- 
line connections, others support new relationships (Boyd & Ellison, 2008; Lenhart & Madden, 2007). Thus, the most 
remarkable feature of the SNSs is the fact that these sites allow individuals to select others with whom they can 
share their interests, activities, and information with. By focusing on a wide range of communication and 
engagement scholars have been conducting empirical research on how SNSs can be better utilized within an e- 
learning environment, especially in higher education (e.g, Hewitt & Forte, 2006; Arnold & Paulus, 2010; Stevenson 
& Liu, 2010). However, faculty or educators have potential dilemmas in relation to the separation between 
schoolwork and the privacy of learners in terms of SNSs utilization in education (Schwartz, 2009). Learners have 
also been found to have a tendency to keep to their personal space rather than allowing others to monitor their 
privacy with instructors since they are using SNSs as a means of popular entertainment (Madge et al., 2009; Selwyn, 
2009; Siemens & Weller, 2011). 
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Mixable as an Education-based SNS 

Mixable is an education-based Social Network Site (SNS) created for the purpose of blending students’ 
social lives with their academic studies. To achieve this end, Mixable has six key features: (a) it connects with the 
student’s Facebook and Twitter account in order to share and create online study groups (as shown in Figure 1); (b) 
it connects with Dropbox for the purpose of sharing images, documents, or videos with classmates; (c) it sends a 
pushing email or notification for Smartphone users to allow students to examine update postings (as shown in 
Figure2); (d) it provides a Smartphone application (app) that allows students to access Mixable regardless of time 
and locations (as shown Figure3); (e) it has three different user options such as connect, follow, and opt-out which 
are intended to secure students’ privacy; and (f) it operates within the university academic management system in 
order to connect to enrolled classes (Tally, 2010). 



mixable 

A social learning environment centered on the classroom 



Connect to classes using devices and sites most familiar to students 



D ===== M 




Figurel. The Introductory Web Page for Mixable 



Based on these six key features of Mixable, the authors have employed two types of support, soft and hard 
scaffolds, in order to explore how an education-based Social Network Site (SNS) supports learners. Soft scaffolds 
are a form of dynamic, situation-specific aid provided by a teacher or peer to help with the learning process, while 
hard scaffolds are static supports that can be anticipated and planned in advance, based upon typical student 
difficulties with a task (Saye & Brush, 2009). 




Figure2. Sample web pages (e.g. file sharing, lecture podcasting, video sharing, etc.) 



In terms of soft scaffolds, Mixable provides several functions. For example, instructors can create and 
sustain sub-networks for common interests related to course subjects in Mixable and can then encourage students 
iteratively to move between individual critical reflection and shared discourse. They can also diagnose 
misconceptions, provide probing questions, comment, and add additional information in an effort to ensure 
continuing cognitive development, and they can further model the critical thinking process. Moreover, instructors 
can provide clear cut answers for questions and evident guidance or feedback for learning activities and then 
encourage students to understand course progress through participation in the students’ dialogues. 
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Figure3. Mixable Mobile 

With respect to hard scaffolds, Mixable allows students to openly communicate with participants regarding 
academic matters as well as individual interests by connecting with Facebook and Twitter. It also allows students to 
create online study groups and to share their individual ideas with other students. Students can then gain access to 
their uploaded files in Mixable in order to view their collaborated works through the Smartphone application or 
Dropbox regardless of either time or location. These types of group cohesion may help students to obtain increased 
perceptions about the learning outcomes from online collaboration (Swan & Shih, 2005). Connections with Dropbox 
lead students to engage in effective sharing strategies. Unobtrusive push notifications help students to check the 
course updates in a timely manner, and this then enhances a feeling of belonging or a “sense of community’’ 
(Garrison, 2007). Mixable also supports the following features: posting with relevant documents and/or video news 
clips to trigger an authentic discussion, showing all updates at a glance from members, and choosing options for 
sharing posts to either the public or the private communities. 

The Community of Inquiry (Col) Framework 

The Community of Inquiry framework emerged from a growing body of literature that examines the sense 
of community for knowledge construction that can be created online (Rovai, 2002; Thompson & MacDonald, 2005) 
and can significantly impact students’ learning processes and learning outcomes (Rovai, 2002; Shea, 2006). The Col 
framework and each of its three constituent elements (social, teaching and cognitive presence is a process-oriented, 
comprehensive theoretical model that allows researchers and practitioners to understand the dynamic nature of 
collaborative learning within online communities (Garrison, Anderson & Archer, 2001; Garrison & Arbaugh, 2007; 
Garrison, Anderson, & Archer, 2010). To sustain an effective learning community, Garrison et al. (2000) have 
addressed the three core elements and contended that the overlapping nature of these three presences creates a 
community of inquiry in which learners can better experience rich collaborative learning. 

Cognitive Presence, as operationalized through the Practical Inquiry model (PIM), is seen as developing 
through a series of four cyclical stages of reflective inquiry beginning with a triggering event and then moving 
(again ideally) onto exploration, integration and resolution (Akyol et al., 2009; Garrison, Anderson, & Archer, 2001; 
Swan, Garrison & Richardson, 2009). Social Presence is defined as the degree to which participants in computer- 
mediated communication feel affectively connected with one another (Garrison & Arbaugh, 2007; Rourke et al., 
1999). In online learning communities, the Social Presence can be examined through the learners’ ability to identify 
with the community, communicate purposefully in a trusting environment, and develop inter-personal relationships 
(Swan, Garrison, & Richardson, 2009). The Teaching Presence, as a critical principle element of the Col model, is 
defined as “the design, facilitation and direction of cognitive and social processes for the purpose of realizing 
personally meaningful and educationally worthwhile learning outcomes” (Anderson et al., 2001). Since the Teaching 
Presence is directly related to the role of the instructor and therefore enhances social interactions and learning 
achievements among the varied participants, a thoughtful, focused, and attentive Teaching Presence is required to 
sustain a successful community of inquiry (Anderson et al., 2001). 

Likewise, utilization of the Col model in online learning has been well-documented (Cleveland-lnnes, 
Garrison, & Kinsel, 2007; Ling, 2007; Akyol & Garrison, 2008; Garrison, 2009; Shea & Bidjerano, 2009). Thus, it 
might be seen as a reliable model to establish effective utilization strategies of social network sites (SNS) and to 
evaluate the sustainability level of online communities by measuring the cognitive, social, and teaching presences of 
the participants. 
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Purpose of Study 



The purpose of this study was to determine effective strategies of SNSs and to better illustrate a 
foundational approach in order to incorporate the Col framework within an exploration of the educational use of 
Social Network Sites such as Mixable. Specifically, the authors have explored how the Cognitive Presence, Social 
Presence, and Teaching Presence from the Col framework and their overlapping features take on a critical role as a 
catalyst to establish and sustain a social network site for teaching and learning within formal education. 
Consequently, we have developed possible hypotheses through the Col framework. 

Research Hypotheses 

Since a review of the literature has shown that the overlap of these three elements provides the structure to 
understand the dynamics of deep and meaningful online learning experiences, we have focused on how the three 
elements of the Col could collectively impact the students’ learning along with the hard and soft scaffolds which are 
addressed in this study. Therefore, the following research hypotheses were tested: (1) the scaffolds designed for 
Mixable enhance students’ social presence which can be examined by the students’ ability to identify with the 
community, communicate purposefully in a trusting environment, and develop inter-personal relationships, (2) the 
scaffolds designed for Mixable support the instructors’ teaching presence, defined as the design, facilitation, and 
direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally 
worthwhile learning outcomes, and (3) the scaffolds designed for Mixable can support higher-order learning in a 
Mixable community of learners engaged in critical reflection and discourse. 

Methods 



Research Context 

The participants for this study were eight graduate students and an instructor of a face-to-face course 
entitled ‘Computer and Video Game Design for Education’. This course had the following features: 1) only graduate 
students could take this course; 2) the instructor used Mixable as the major platform to share the course information 
with students; and 3) students were encouraged to use Mixable for their group discussions. An exploratory study was 
utilized to provide broad information related to educational scenarios and contexts within Mixable. This exploratory 
approach is not only used to better understand the context, but has also been undertaken in order to analyze data 
through extensive interviews or observations (Patton, 2002). Demographic data of the participants, such as age and 
gender, were also collected along with data from the Col survey instrument (Swan et al., 2008). All study 
participants were between 25 to 35 years of age and were mostly males (62.5%). 

Data Collection 

Community of Inquiry (Col) Survey. The Col Survey Instalment (Swan et ah, 2008) was administered to 
students to gather data using an online survey (see Appendix A). The 34 Col survey items were measured on a 5 
point Likert scale (Strongly Disagree = 1 and Strongly Disagree = 5). The standard multiple regression was used to 
predict how cognitive presence is shaped and predicted by teaching and social presences. 

Interviews. . Follow-up student interviews were administered to the students by email at the end of 
semester. The interviews focused on information regarding the students’ satisfaction with Mixable as an 
instructional tool, the student’s view of the benefits and challenges of Mixable as an education-based SNS, and 
suggestions for Mixable. (e.g., How can the instructor foster students to share enriching learning materials in 
Mixable? and How can a sense of collaboration be developed in Mixable? 

Online Observations. Online interaction transcripts between the instructor and the students and among the 
students in Mixable were gathered. To prevent possible interference, the study authors did not participate in any 
interaction but observed the students’ online participations and interactions in Mixable. A virtual observation was 
also implemented once a week. In this observation, authors focused on how the instructor and the students actively 
participated in online activities and interacted with each other in Mixable. 

Data Analysis 

Survey > Analysis. Since six students out of eight participants responded to the online Col survey and the 
dependent data was not normally distributed, the Mann- Whitney Test was conducted in this analysis. The influence 
of the teaching presence on the sustainability of social and cognitive presences was examined. Moreover, the 
relationships were determined among the three elements of Col (i.e. such as the teaching, cognitive, and social 
presences) within Mixable communities. 

Interviews and Observation Analysis. Content analysis was performed for both interviews and observation 
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analysis. To use the content analysis, the authors coded each transcript in both interviews and observation into 
conceptual chunks and into categories according to the underlying research hypotheses. Then we combined those 
categories under a certain type of presence in order to identify the relationships. Finally, we created assertions by 
merging the quantitative and qualitative data. 



Results 

HI. The scaffolds designed for Mixable enhance students ’social presence that can then be examined by the students ’ 
ability to identify with the community. 

The Mann- Whitney Test was conducted using data from the online Col survey to examine the influence of 
the teaching presence on social presence. However, table 1 showed that there was no significant influence of the 
teaching presence on social presence in the course of this study (U = 3.500, z = -.443, p = .658). 

Table 1: Mann-Whitney test statistics for the influence of teaching presence on social presence. 





Social Presence 


Grouping Variable 


Teaching Presence 


Mann-Whitney U 


3.500 


Z 


-.443 


Asymp. Sig. (2-tailed) 


.658 


Exact Sig. [2*(l-tailed Sig.)] 


.700“ 



a. Not corrected for ties. 



This is not surprising given that the course of this study was offered in a blended learning format which 
implies that technology was “bolted” onto a traditional course, using technology as an add-on to share learning 
resources and to facilitate student-led learning activities. As a result, students acknowledged that Mixable was 
effectively used as a tool to achieve the course goals as a knowledge sharing space. However, they also indicated 
that Mixable may have the potential to create a more effective social learning community. For example, one 
interview participant reported, “Judging from other peoples' posts, you can get an idea of their personality. But most 
of the interaction and bonding occurred in class. . . we used Mixable mostly for updates of what we had already 
planned. (Group 1)” Similarly, other participants said, “Probably in online classes, or classes with less time for this 
type of activities during class time, this would change (Group2)”, “This does apply to this course. The interpersonal 
relationships were developed in our classroom environment (Group 1)”, and “Since this was a classroom based 
course, using mixable for reflection and discussion of course readings might be helpful. (Group3)” With regard to 
this constraint within a bounded course environment, most participants appealed that they would like to use Mixable 
differently in the future. For example: 

“I would like to have specific purposes for Mixable. (Group2)” 

“Mixable is the most sociable teaching tool I've ever known so far. So I’d like to use it for collaboration and 
discussion. This would have to be structured and outlined prior to use during the course. (Groupl)” 

“I would like to facilitate interaction more effectively. (Group3)” 

As Garrison and Vaughan (2008) have emphasized, “Blended learning is the organic integration of 
thoughtfully selected and complementary face-to-face and online approaches and technologies (p. 148).” Unless an 
instructor intends to fundamentally redesign teaching and learning approaches in ways that realize increased 
effectiveness, convenience, and efficiency, any type of emerging technologies cannot maximize its potentials for 
educational purposes, such as engaging students in critical discourse and reflection. In other words, instructors need 
to redesign the way that courses are developed, scheduled, and delivered using Mixable in order for their teaching 
presence to better enhance the sustainability of the social and cognitive presences by a significant determinate of 
student satisfaction, perceived learning, and sense of community. 

According to our observation analysis, most students in this study felt comfortable exchanging questions 
and answers about the course clarifications or struggles that they felt necessary to uncover. For instance, when they 
had a question about the readings assigned, they tried to share the question rather than sending email to the 
instructor directly so that they would get instant feedback from other students or from the instructor. Even when they 
had a problem with downloading files connecting to the file sharing program, they suggested solutions to each other 
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rather than merely waiting on directions from the instructor. Similarly, students were willing to share their personal 
experiences related to the course. 

Some students made comments regarding lack of interaction between instructor and students in 
Mixable. As their interaction mostly happened in the classroom, students described their use of Mixable on 
receiving updates and course material in practical ways. 

“Mixable is a bit impersonal. Everyone introduced himself/herself on the first days, so that helped. Also, 
judging from other peoples' posts you can get an idea of their personality. But most of the 
interaction/bonding occurred in class. It might have to do with the nature of the class, as discussions are 
easier done in person, and we used Mixable mostly for updates of what we had already planned. Probably 
on online classes, or classes with less time for this type of activities during class time, this would change. 
(Group 2)” 

“The interpersonal relationships were developed in our classroom environment. (Group 1)” 

“Well, the pictures helped a lot. But I don’t think Mixable helped us to have any kind of dynamic 
interaction in there. (Group 3)” 

Although students tried to seek an understanding of the course activities, they did not engage in higher 
levels of knowledge construction. It might be caused by a limited utilization of Mixable focusing on sharing files, 
announcements, and Q&A rather than their choice to employ online discussion which could better help the students 
to learn and communicate in depth. 

H2. Scaffolds designed for Mixable support instructors’ teaching presence defined as the design, facilitation and 
direct instruction. 

Research on teaching presence has demonstrated that teaching presence is a significant determinant of 
learner satisfaction, perceived learning, and sense of community. Specifically, a number of empirical studies has 
demonstrated that since teaching presence is related positively to social presence (Shea et al., 2006; Gilbert & 
Dabbagh, 2005), in the absence of teaching presence, student discourse is impoverished (Finegold & Cooke, 2006; 
Meyer, 2003). This study has also examined how teaching presence may affect either social or cognitive presences 
with the results of the online Col survey. However, according to Table 1 and 2, no significant effect of the teaching 
presence was found with regards to social presence ( U = 3.500, z = -.443, p = .658) or cognitive presence ( U = 
3.000, z = -.655, p = .513) in this study. 

There was little evidence that Mixable was used as an effective facilitation tool to allow students to realize 
that they could achieve personally meaningful and educationally worthwhile learning outcomes in the teaching 
presence. For example, one participant mentioned, “Well, Mixable was a practical way on receiving updates and 
course material. However, I don’t think it was used as an effective facilitation tool. It does not apply to this course.” 
Although another participant noted, “Firstly, we got to know its updates (in Mixable) via email, so we could check it 
timely. Secondly, the instructor responded to students' questions or concerns timely,” she didn’t agree that the 
instructor actively used Mixable to facilitate students’ ability to engage in student-led social networking and/or 
knowledge creating. These perceptions were most likely the result of the blended learning environment employed in 
this course. In terms of our observation on posts, since the instructor’s particular focus on Mixable was not on online 
discussion, the instructor seemed to focus on providing clear guidelines and reminders in order to support face-to- 
face class discussions throughout the course. Mixable was usually employed as a communication tool. As displayed 
below, the instructor not only announced important due dates for student assignments but also clearly guided the 
class towards understanding learning activities. 

Instructor Reminder: Your first assignment is due in two weeks. I have attached the description for it here. 
Also, for next week, please come to class with a brainstormed idea for your group's educational game. This 
doesn't have to be formalized, but should be a completed concept document, that describes game goals, 
player activities, what a typical 30 seconds of gameplay would be like, what your learning objectives are, 
what the hooks for the game will be (things that engage players). Finally, be sure to have completed your 
D&D character sheet (at least the class, name, race, & attributes) as we will start with running a short game 
session. 

While the instructor’s direct instruction or feedback was rare within the Mixable platform, there were 
prompt responses to students’ questions or concerns. In particular, the instructor tried to provide immediate course 
clarifications so that the students could proceed more effectively with their work 
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H3. The scaffolds designed for Mixable can support higher-order learning in a Mixable community of learners 
engaged in critical reflection and discourse. 

Paramount in the Col framework is the construct cognitive presence, which Garrison et al. (2000, p.12) 
conceptualized as the extent to which learners are able to construct and confirm meaning through sustained 
reflection and discourse. Therefore, the effects of teaching and social presences on cognitive presence were 
examined with the online Col survey data. However, as Table 2 showed, no significant effect of teaching (U — 3.000, 
z = -.655, p = .513) on social presence ( U = 3.000, z = -.655, p = .513) on cognitive presence was found. 

Table 2: Mann-Whitney test statistics for the effect of teaching presence and social presence on cognitive presence. 



Cognitive Presence 


Grouping Variable 


Teaching Presence 


Social Presence 


Mann-Whitney U 


3.000 


3.000 


Z 


-.655 


-.655 


Asymp. Sig. (2-tailed) 


.513 


.513 


Exact Sig. [2*(1 -tailed Sig.)] 


.700 a 


.700“ 



a. Not corrected for ties. 



Students appreciated the effectiveness of Mixable to permit prompt communication and trigger events for 
further inquiry; however, they didn’t agree that online activities in Mixable were valuable for higher levels of 
engagement. For example, when students were asked if Mixable helped them appreciate different perspectives, one 
participant mentioned, “We got into the habit of using Mixable because it was the default way to communicate with 
the instructor. It was an efficient way to communicate with team members and having it within one location (not 
having to browse through mails).” Similarly, another participant pointed out that the usefulness of Mixable in this 
class is limited to the replacement of the previous learning management system, noting, “I'm not sure I learned 
anything skill-wise in particular, as the way it works is similar to Facebook and it has a similar functionality to other 
tools, such as blackboard vista. In contrast with blackboard vista, having a notification system encouraged me to use 
the tool more. I rarely visit my blackboard account, and I used Mixable much more, greatly because I knew when 
new content was available.” Some participants noted that there were little newly gained perspectives in terms of 
collaboration in teams via Mixable. The ability to take the perspectives of others is among the most important skill 
for those who want to actively engage in social network sites. Similarly, transformative learning through emerging 
technologies entails becoming more aware of other perspectives and ideas. Therefore, we have expected that 
students could explore the issue, both individually and corporately, through critical reflection and discourse via 
Mixable, but little evidence to this extent was actually detected. In terms of this issue, one participant noted, “The 
instructor didn’t foster the students to do so, but the students did it actively”, and then suggested that instructors 
should foster students to share enriching learning materials and relevant opinions (perspectives) in Mixable. 

According to the analysis of students’ introduction posts, most students were highly motivated to take the 
course. They commonly described that they are taking the course with an increased interest in educational game 
design, which is the main course content, and also with an interest in games themselves. Moreover, these students 
engaged in sharing their initial ideas of educational game design and final products respectively at both the 
beginning and end of the course, even though they did not comment on their ongoing progress during the process. 

Students did not exchange feedback on their projects with each other in Mixable because these projects 
were already presented in class. However, Mixable was likely to be an efficient place to review their outputs and 
further reflect on the course. Students reported in their reflection postings that they could develop their ideas by 
combining these with the new ideas or perspectives from other students. In addition, they tended to utilize Mixable 
to bridge the distance between class and outside of class meetings; but they did not have a dynamic interaction or 
engagement in Mixable. For example, they tried to upload documents that they had already presented in class with 
short descriptions. Thus, Mixable could be a repository for knowledge-sharing as an extension of class work. 

Conclusions 

The result from this exploratory study allows us to confirm that instructor’s intention and course design are 
crucial prerequisites for a successful blended higher educational experience. As widely known, since blended 
learning does not mean a simple combination of face-to-face and online, instructors need to redesign course 
materials and/ or tools in order to integrate the best features and in-class teaching with the best features of online 
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learning to better enhance student-led active knowledge creation, sharing, and improving. Consequently, this study 
emphasizes the importance of the teaching presence in creating and sustaining social and cognitive presences in 
online learning environments. This argument is echoed by Shea and Bidjerano (2009) who claim that there is the 
central role which the teaching presence plays and then go on to provide important insights into how best to 
integrate the constituting elements (i.e., presences) of an online community of inquiry. 
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Abstracts 

The transformative potential of virtual communication argues for extending interdisciplinary collaboration. 
Due to the complexity of the topic, there is a need to establish a common base to understand the emergence and 
sustainability of virtual research collaboration, informally between networks of colleagues and formally within 
established teams and organizations. Thus, the authors developed a framework addressing virtual collaboration 
effectiveness for interdisciplinary research and choose a new emerging field of engineering education research as 
the context. 



Introduction 

Much scholarly work has been done on the topics of virtual team, but there is little consensus among 
researchers on the holistic and working dynamics for the sustainability of virtual organizations. There have been 
several comprehensive reviews about how to promote collaborative performance capabilities within entrepreneurial 
situations focusing on individual perceptions about team communication (e.g. leadership, trust, conflict management, 
etc.) (Jarvenpaa & Leidner, 1999; Hollingshed et al. 1993; Steinfeld, 2002). These works are mostly based on the 
classic works of researchers such as McGrath (1984), Hackman (1987), and so on, who expressed the nature of team 
performance following the systems theory in which inputs lead to processes, which in turn lead to outcomes: Input- 
Process-Output (IPO) model. The IPO model explains the causal effect in a linear direction (e.g. the process can’t 
influence the inputs). However, virtual collaboration is complex adaptive systems, in which processes also influence 
the inputs over time, and the relationship is not linear. Therefore, this type of linear approach is difficult to capture 
the full dynamics of virtual collaboration highlighting the reciprocal interaction of human actors and structural 
features of organizations over time, especially in the context of research enterprises. 

For any interdisciplinary discipline, it is critical to chart its birth and growth to understand where the 
discipline stands and what innovative endeavors lead to the creative accomplishments currently witnessed in its 
knowledge products. But, it is challenging to have well-described bodies of knowledge for their research, and an 
accepted terminology and vocabulary. Because, first of all, neither there is consensual agreement on the definition of 
interdisciplinary research, nor are there widely recognized, valid, and reliable measures of interdisciplinary research 
activity or output. Even what to consider as the “disciplines” among which interdisciplinarity would occur is messy. 
Then, how do we know when large-scale interdisciplinary research collaborations are happening? Also, how do we 
know that research utilizing large datasets attracts a large number of researchers to utilize these datasets? Can we 
take a data-driven approach to clearly point out trends in interdisciplinary research productivity and collaboration? 

To explore these ideas, it is attempted that a theoretical base is established to empirically investigate the 
evolution from loose network into more purposeful virtual organizations in one particular interdisciplinary setting: 
engineering education research (EER). EER aims to produce empirical results to advance the teaching and learning 
of engineering by conducting interdisciplinary research. According to the call for radically rethinking education 
research to include large scale data and collaborations, there is an urgent need to understand the boundaries of the 
EER field, its contributors and, structures and processes underlying the collaborations between researchers. In 
response to the urgent need, the focus of this paper was on ‘virtual research collaboration’ and ‘interdisciplinary 
knowledge producing interaction’ within EER. 



Literature Review 



Scientific Collaboration 

Collaboration plays a critical role for scientific research, which is dominated by complex problems, rapidly 
changing technology, dynamic growth of knowledge, and highly specialized areas of expertise. Scientific 
discoveries are considered collaborative products of scientists working together rather than in isolation. 6 Scientific 
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collaborations are thus simultaneously shaped by social norms of practice and prevailing knowledge structures as 
well as available and evolving technological infrastructure within and across scientific disciplines. Scientific 
collaboration has been defined in the bibliometric literature as the set of interaction taking place within a social 
context among two or more scientists that facilitate the sharing of meaning and completion of tasks with respect to 
mutually shared, super ordinate goals. 6 ' 7 For research on scientific collaboration, documents are considered as 
valuable sources to constitute a core knowledge repository as they detail important scientific investigations or 
development efforts, and report key experiments or analysis results. 

There are three major streams using these documents in terms of the analysis of scientific productivity: text 
mining, information visualization, and network analysis (Chen & Paul, 2001). Text mining extracts important 
relationships or patterns from a collection of textual documents, and evaluates and interprets those patterns (Chen & 
Chau, 2004). It reveals important subjects or topics embedded in the title, abstract, or main body of documents. 
Information extraction is capable of extracting important entities of interest from structured texts effectively and 
efficiently. The authors documents on the basis of author, institution, topic area, and region, and analyzes them to 
identify important themes, patterns, or trends (Chen & Paul, 2004). Network analysis is also central to knowledge 
mapping as it can be used to segment subgroups of scientists and researchers, identify key people in a network, 
reveal their interaction (e.g. collaboration) patterns, and depict the overall network organization or structure (Chen 
& Paul, 2004). Several essential measures have been developed to characterize each individual node’s role in a 
network; e.g. degree, betweenness, and closeness (Wasserman & Faust, 1994). The degree of a node describes the 
number of direct links it has. The betweenness of a node depicts the number of geodesics, i.e., the shortest path 
between any two nodes passing through the node. And the closeness of a node denotes the number of all the 
geodesics between that node and every other node in the network (Hu et al., 2011). 

Organizational Network Analysis 

To understand the overall knowledge landscape, the authors empirically explored scientific collaboration in 
the EER at the different levels; the individual level, the group level, the organizational level or at the level of the 
scientific institutions. Classifying engineering education research organizations into specific categories on the basis 
of organizational structure presents a challenge to existing theoretical frameworks on account of the multiple levels 
of analysis at which network and other organizational structures may be characterized within the trans-disciplinary 
domain of engineering education. 

In order to research the emergence of a larger community we have to take into account that networks and 
organizational structures do not fit neatly into established schema provided by established literature on virtual 
organizations and network structures. Organizational structures within the domain of engineering education include: 
(a) Intra-university organizational structures such as engineering education departments (e.g. at universities such as 
Purdue and Virginia Tech) and other organizations which exist at the intra-university level (e.g. the Engineering 
Education Research Center (EERC) at Washington State University and the Institute for P-12 Engineering Research 
and Learning (INSPIRE) at Purdue), (b) Inter-university organizations which exist as collaborations between 
researchers across U.S. universities (e.g. the Center for the Advancement of Engineering Education (CAEE) and the 
Center for the Integration of Research, Teaching, and Learning (CIRTL), (c) Network organizational structures 
which emerge and exist at the national and international level (e.g. the ASEE and its various divisions, SEFI, IGEP, 
and REEN). 

Given the breadth of organizational structures which exist within the EE and EER domain the central 
problem from an analytic perspective is to employ theoretical constructs relevant to the structure of organizations, 
and the process of organizing. Such constructs will provide a unit of analysis (such as a team, a virtual organization 
or a network) that is meaningful and relevant to an analysis of the emergence and sustainability of various types of 
organizations within the domain of engineering education. Given the need for studying organizational structures 
flexibly across multiple levels of analysis, we propose to study the emergence and sustainability of these 
organizations using four distinct conceptualizations of organizational structures and the organizing process. 

Personal Network Analysis 

The personal network analysis in our research focuses on the collaboration in engineering education 
research and the components included in our analysis follow Chiu, Hsu & Wang (2006)’s knowledge sharing model, 
which is based on the social cognitive theory and social capital theory. The social cognitive theory defines “human 
behavior as a triadic, dynamic, and reciprocal interaction of personal factors, behavior, and social network (system)” 
(Chiu et al., 2006: 1873). According to the gist of this theory, personal outcome expectations and community-related 
outcome expectations are the factors that influence human functions (Chiu et al., 2006). 
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In our research framework, the personal outcome expectations and the community-related expectations 
make up the first part of the personal network analysis. The personal outcome expectations refer to a member’s 
judgment of the likely consequences that his/her membership will bring to himself/herself. The community-related 
outcome expectations refer to a member in a virtual engineering education research and collaboration’s judgment of 
the likely consequences that his/her membership will bring to the organization. 

The second part of the components in the personal network analysis complements the outcome expectations 
to measures the influence of social networks of virtual engineering education research communities on individuals’ 
behavior. The components herein are based on the social capital theory. As social capital is the “features of social 
organization such as networks, norms, and social trust that facilitate coordination and cooperation for mutual benefit” 
(Putnam, 1995: 66), it is a relevant notion to virtual communities because it indicates an abstract hidden source that 
can be tapped when members in a virtual community interact and collaborate with each other (Chiu et al., 2006; 
Daniel, McCalla & Schwier, 2002). Based on our review of the social capital literature, we will be employing the 
following operational definition for social capital in virtual engineering education research communities: a common 
social resource for people conducting engineering education research to support information exchange, knowledge 
sharing, and knowledge construction through interaction, built on shared understanding and emotional connection, 
trust, mutuality, professional identity, subjective norm and maintained through social participation, plurality and 
social protocols. The following table demonstrates the operational definitions of components included in our 
framework: 

• Social Interaction Ties: Strength and depth of the relationship and communication frequency among 
members in a virtual scientific research organization and collaboration. They are the channels for 
information and resource flows in virtual communities where members seeking for knowledge-based 
communication (Chiu et. al, 2006; Peters & Manz, 2007; Chen et. al, 2009; Tsai & Ghoshal, 1998; 

Nahapiet & Ghoshal, 1998; Wasko & Faraj, 2005) 

• Shared Understanding: A member’s knowledge of the strategic direction of the virtual community; other 
members’ roles, tasks, responsibilities, specific expertise and needs; research tasks; and the virtual 
environment of scientific research (Peters & Manz, 2007; Schwier & Daniel, 2006) 

• Shared Emotional Connection: Shared history and crisis, investment of time and resources, honor and 
humiliation, and spiritual bonds among members of virtual scientific research organization and 
collaboration (McMillan & Chavis, 1986) 

• Trust: A member’s expectation that other members in a scientific research organization and collaboration 
will follow a set of values and principles. It is the level of certainty that one community member uses to 
assess the action of another member (Chiu et. al, 2006; Schwier & Daniel, 2006; Peters & Manz, 2007). 

• Mutuality: A members of a virtual scientific research organization and collaboration construct purposes, 
intentions, and the types of interaction interdependently. The information exchanges and the research 
collaboration are mutual and reciprocal among members (Chiu et. al, 2006; Schwier & Daniel, 2006) 

• Professional Identity: A member’s sense of belonging to a virtual scientific research organization and 
collaboration and his/her positive feeling towards the community (Chiu et. al, 2006). 

• Subjective Norm: A member’s perceptions of whether the behavior is accepted, encouraged, and 
implemented by virtual scientific research organization and collaboration (Chen, Chen, & Kinshuk, 2009). 

• Social Participation: A member’s social participation in a scientific research organization and collaboration, 
especially the participation that sustains the virtual community (Schwier & Daniel, 2006). 

• Plurality: A member’s “intermediate associations” such as families, friends and other peripheral groups that 
he/she uses to enrich the virtual scientific research organization and collaboration (Schwier & Daniel, 

2006). 

• Protocols: Social protocols refer to the rules of engagement, acceptable and unacceptable ways of behaving 
in a virtual community of scientific collaboration (Schwier & Daniel, 2006). 

Preliminary Results 

Building on prior theoretical and methodological insights from social studies of science and bibliometrics, 
we address here a keyword-based scheme to identify significant trends and patterns in EER. We selected K-12 
Engineering Education (K-12 ENE) as a sub-discipline in the EER research arena. Most K-12 ENE relevant 
publications have been published in the past 10 years, indicating that the field of K-12 EER is relatively new. The 
authors conducted a bibliometric analysis to discover the emergence and development of this sub-discipline by 
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establishing a keyword-based scheme. The keyword-based scheme offered the benefit of defining K-12 EER in 
terms of its vocabulary and offered a universal approach that is currently primarily used for most electronic sources. 
And this bibliometric data gathered using keywords was used to conduct a field analysis and a social network 
analysis. 



■ teac* strateg* 

■ partnership 

■ interdisciplin* 

■ communicat* 

■ STEM (Science Technology 
Engineering Mathematics) 

■ innovat* 

■ curricul* 

■ techn* literacy 

Figurel. Top Ten Keywords in Publications 1980-2010 
Keyword Creation/Refinement/Classification 

In order to begin, it was necessary to create a list of keywords by using the 66 keywords from the literature 
(e.g. Katehi, 2009, 2010). The finalized keyword sets are: K-12, kindergarten, elementary school, middle school, 
high school, pre-college, children, and P-12. These words also came from the previous two sources. Next, all the 
keywords were put into three online databases, Web of Science®, Compendex®, and EbscoHost®. These searches 
also allowed for the addition of words such as, advocat*, student engag*, cooperat*, implement*, facult*, etc. A list 
of 43 selective and effective (meaning on which produces relevant results) keywords was established. Then, the 
frequency of each keyword was split into three columns based on decades. These keywords were also divided based 
on the classification defined earlier; as a result was obtained. Figure 2 shows the cumulative distribution of 
keywords into the aforementioned categories for the entire duration of 1980-2010. 



Keyword Classification Distribution 




A • C O I f C 



A. learning mechanisms 

B. epistemologies 

C. learning systems 

D. educator's practices/curriculum 

E. diversity and inclusiveness 

F. competence 

Figure2. Keyword Classification Distribution 




Keyword Analysis 

The analysis of bibliometric data was first done via keyword. An excel file was created with the keywords 
down the first column and the years broken down into periods of three years from 1980-2010. In addition to the 
keywords, the narrowing words, teachers, counselors, and pupils, were added to the keyword column. The Web of 
Science® database sub-database was sorted by year in ascending order. The ‘Keywords’ section of the WoS 
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database was filtered via ‘Contains. . . ’ for each individual keyword along with the narrowing words. The number of 
times each word occurred per decade was calculated by Excel and manually inputted into the new Excel file in the 
corresponding keyword and year range cell. This information was used to create figures 1 and 2. 

Social Network Analysis 

NodeXL was used for the analysis of author contributions. All co-author relationships were mapped 
manually. For example, John, Jane, and Grace are authors. As shown in Figure 2, the relationships would be as 
follows: (1) John-Jane, (2) John-Grace, and (3) Grace-Jane. Single authors were mapped as self-loops (i.e., Bob- 
Bob). The co-author relationships were inputted into NodeXL®. The data was then mapped. The overall metrics 
were calculated via ‘Overall Metrics’. Due to the density of the map, it was necessary to choose the top ranked 50 
authors based on their degree (i.e., the authors that had the highest degree were used). The map was condensed to 
these authors by used the ‘Dynamic Filter’ tool in NodeXL® to choose the degree minimum and maximum (11-23). 
The data was then analyzed manually. 




Figure2. Social Network Map of K-12 ENE 



Research Productivity 

The keyword analysis showed that over the past 10 years, there is a spike in K-12 ENE research producing 
most research and publication in this area. The most frequent keywords are educat*, STEM, stud*, teach*, and 
curricul*. But, at the same time, the K-12 ENE research began with research that had a very narrow perspective. 

For example, while in 1980s little or no research had been done in the area of K-12 engineering education and in 
1990s research in this area began covering 13 keywords, the research in the 2000s rose dramatically to encompass 
42 keywords. Another example is that, whereas there were only 25 keyword occurrences in publications in the 1990s 
on K-12 ENE in WoS, in the 2000s, this number expanded to 736 keywords occurrences. The keywords and the 
decades they occurred in has the potential to reveal current trends in K-12 EER. However, this is up for 
interpretation. The progress of this new field as a whole can be located in figure 3. 



Number of Keyword Relevant Results vs. Decade 




Figure3. Number of Keyword Relevant Results vs. Decade 
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Limitations 



Limitations in this specific research project include that only one database, the Web of Science database, 
was used. This is an ongoing research project and more databases will be analyzed later. Also a large limitation is 
the possibility that 43 keywords did not capture all K-12 ENE relevant articles from the pre-existing database. 
However, to cover more articles a larger list of keywords with greater variety would be needed. This could be done 
by using a greater variety of literature to develop the keywords. An alternate solution would require checking a very 
large amount of publications (at least thousands) manually for any correlation. This solution was not time effective 
for the timeline of this project. So it was assumed that the keywords did capture all relevant articles from the pre- 
existing database. In addition to this, for the demographic analysis a total of 494 articles were captured in acquiring 
the data. However, it is assumed that this analysis is still accurate, because the other articles could be general to K- 
12 ENE as a whole. 



robotics desirn^T 1 

outreach • ^..school- stem • systems ^ 




Figure4. Word Cloud from the Publications 1980-2010 



Conclusion 

The bibliometric analyses show that engineering education research began in the last 10 years and has 
developed to cover a greater number and variety of keywords in the more recent years, which indicates how the field 
has reached a high level of academic standard. The word cloud (figure 4) reveals that the most popular words were 
engineering, education, learning, science, school, STEM, outreach, K-12, children, development, design, robotics, 
gender and many more. 

The classification and demographic analysis show that most researchers have an interest in high school 
ENE. For this reason, it is safe to assume that many believe high school might be the most vital to the 
implementation of K-12 ENE. Furthermore, epistemologies and learning mechanisms have been the most widely 
researched. This may reveal that the field is moving toward implementing K-12 ENE standards, which is why the 
class of educator's practice curriculum has been so neglected. The groundwork has been set and now the research on 
an ENE curriculum will begin. 

This reveals that the field of Education, Scientific Disciplines is looking into the benefits of K-12 ENE on 
their field. This may be the field that will be most impacted and it may be the field that many standards relate to due 
to the large amount of researchers in this field. 

The social network analysis results reveal that there is a high trend of collaboration between authors in the 
K-12 EER community. Krause, S. was found to be the most collaborative author — who also had a significant 
position as a bridge for communication for other authors in this field in addition to Roberts, C. The fact that the two 
most collaborative authors are connected is a good sign; however, more collaboration could greatly improve the 
field as a whole. Overall, more work needs to be done in this field in order to make more progress. 
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An Iterative 4 Step Experiential ID Process (4xEID) 



Robert L. Appelman, PhD. 

Abstract 

This paper lays out an Instructional Development process specifically targeting the design of complex experiential 
learning environments such as those found in face-to-face (teacher-student) or virtual game contexts (virtual 
environment-student). A virtual game named iKIDS: Slices in Time focusing on STEM learning is used for 
illustration. This 4-Step Experiential ID Process (4xEID) uniquely integrates content hierarchies, learning level 
taxonomies, teaching methodology, storyline development, experiential mode analysis, and offers a unique 
incorporation of learning vectors. The iterative nature of the process allows for a vector analysis of both the design 
and of learning during game play. 



Outline of the 4xEID Process 

Key to developing an Immersive Learning Environment (ILE) is the adherence to a specialized 4-Step Experiential 
Instructional Design Process (4xEID) . In the case referred to in this document the ultimate outcome of the design is 
envisioned to be a hybrid Board Game combined with a 3D game-like virtual space. This learning environment is 
envisioned to be incorporated in-class with groups of students, or played outside of class on an individual learning 
basis. As the players move through the mechanics of game-play on the board, they will encounter options to also 
interact with a computer that provides responses and scenarios relative to what the player inputs into the computer. 
The reasons for our selection of such an environment is to offer the levels of dynamic design challenges to 
appropriately illustrate the 4xEID process, and to offer adequate access to content resources. The 4xEID process 
could just as well be used to aid students learning how to interact with customers in a store environment, to learn 
strategies and techniques of playing soccer, or to learn any other decision, or skill, based content. 

The 4xEID process begins as does ADDIE and many HPT procedures: 

1. Define the Goals of the learning environment and Bound the Specific Content 

All learning design begins with the objectives of the individual student, a group of students, the needs of a teacher to 
meet curricular goals, or the goals of an institution or organization. A thorough analysis will list goals for each level 
of the hierarchy. For instance, the organization may simply be focused on meeting standards for every student, 
whereas the teacher’s goals may be to both meet standards, but to have the student also apply knowledge. Such a 
difference could create a tension between the teacher and administration relative to the amount of time devoted to 
the content selected. Even the amount and which content is selected may be contentious, so a careful and honest 
analysis at this stage is critical upon starting the design process. 

2. Operationalize the Content : Creating a Content Hierarchy targeting Learning Levels 

Experiential Learning involves hands-on interactions by the student that should trigger specific levels of cognition 
and actions. In other words, the content must be arranged such that it is evident which chunk is most coupled to the 
goals laid out in the previous step. Such an exercise involves not only prioritization of content, but also the targeting 
of the level of cognition and specific action requierd to meet that goal. Fortunately Bloom and Krawthwhol 
developed a Master Design Chart (MDC) that provides an effective worksheet to accomplish both tasks at the same 
time (see Figure 1). In short, chunks of content are placed in one spreadsheet column while numbers of importance 
are placed under column headings matching the levels of cognition or action desired (D. Jonassen, Tessmer, & 
Hannum, 1990; D. H. Jonassen, Hannum, & Tessmer, 1989) 

• Recalls data or information 

• Understands the meaning and interpolation of a problem 

• Applies a concept in a new situation 

• Analyzes component parts, facts, and inferences 

• Synthesizes a new instance of a structure or model 

• Evaluates existing data, relationships, or models 
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Figure 1 

Master Design Chart for STEM content 



By coupling a Learning Level to each Learner Task, it will allow for the design of appropriate level challenges for a 
learner as they progress through the content within the Learning Environment (LE). For example, as learners first 
encounter information or problems, they will need to operate mostly within the first two levels of the learning 
hierarchy, i.e. they will be primarily identifying, recognizing, distinguishing, and possibly selecting an appropriate 
example that would predict success as they know it. Through both access to resources that educate them or through 
failure, they may move up the hierarchy to more complex tasks. Just how far up this hierarchy a learner will 
progress will be a factor of the skill of the learning environment designer, and also the intelligence and motivation of 
the learner. Identifying the skill of the LE Designer is often left out of the equation, but just as in any teaching 
environment there is a wide variance of teaching styles, epistemology embraced, methodology chosen, and 
implementation of that methodology (Hannafin & Hill, 2005). The MDC aids the designer who wrestles with a 
multitude of pedagogical decisions by keeping the learning goals in the forefront of the design decisions. As is often 
the case, the teacher or LE Designer wants to jump to the implementation phase and act intuitively based on what he 
or she “knows”. Such an approach does not embrace a learner-centered epistemology because unless one knows the 
specific capacity of the learner to comprehend each chunk of information, the process of teaching a body of 
knowledge can only offer a “hit-or-miss” result. This is especially true for virtual learning environments that are not 
face-to-face. In face-to-face environments adjustments may be made through interaction with students that offer the 
option of mid-stream corrections (Thiagarajan, 1994). For instance, as Gagne points out (Gagne, 1968; D. H. 
Jonassen et al., 1989), a content hierarchy may be constructed that allows for the appropriate sequencing of content 
for the learner (see Figure 2). 

Because the operationalizing of content involves such micro pains-taking decisions, many teachers and designers do 
not have the patience to engage in this process. However, for complex learning environments that are virtual, i.e. 
programmed, step two is an essential step that allows the designer to layout a very workable environment that meets 
the instructional and engagement goals. Once the goals and content are defined, a very iterative, brainstorming 
process begins where the design process itself is immersed in replications of the learning experience being created. 
The iteration and immersion is the primary difference between typical ID approaches and this EID approach. 
Constant observation of the interactions with content and other collaborators within the LE allow for revisions of 
priorities of the MDC and the methods used in implementing it. 
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Content Hierarchy 



3. Define Specific Methodologies that establish overall Treatment of content and student Interactions 

Because the Master Design Chart is created in a spreadsheet that has a total column at the right (see Figure 1), it is 
possible to sort on the totals column resulting in the most desirable content acquisitions appearing at the top of the 
spreadsheet. In other words, the content is now prioritized. The design process begins with these prioritized chunks 
by determining what tasks students could perform that would demonstrate acquisition of the concepts, procedures, or 
principles. Brainstorming a variety of tasks that meet these goals will allow for certain tasks to resonate with each 
other and allow for a “treatment” to emerge. Treatments are often referred to as storylines or learning contexts, but 
multiple treatments should be pursued through the first couple of iterations of the design process to assure that when 
content is added to the scenarios of the learning experience the selection chosen achieves the most effective results. 
The goals are not only to achieve the highest cognitive functionality, but also to create an engaging learning 
environment. I must clarify here that when I use the word engaging, I do not mean “fun”. Engagement to me means 
that the student is focused and involved in working through the content tasks found in the LE. For a “Serious Game” 
I always presuppose a “Serious Student” who is engaged with the game for learning purposes and not a “gamer” 
who is primarily seeking entertainment. The key is in picking the right storyline and/or context for the LE such that 
the student will be engaged (Csikszentmihalyi, 1990). 

For illustration purposes, we will use a storyline example involving a multi-player game called iKIDS: Slices in 
Time. This game allows each player to negotiate their game -play strategies with other players as they seek out each 
other’s strengths in Physics, Earth Science, Math, or History. During game-play their character will need to choose 
appropriate solutions to questions or strategy problems in collaboration with the other iKIDS. In this STEM learning 
game the content problems would target the problems of "Free Falling Objects" and would require that the learning 
environment allows for “time travel” plus an encounter with Aristotle in Athens Greece, Galileo in Pisa Italy, and 
Sir Isaac Newton in Cambridge England. This complex learning environment allows for the basic physics of falling 
objects to be encountered, and it also allows for the historical development of scientific inquiry to be encountered as 
well. Situating the learning in the authentic historical context that the scientist derived his or her conclusions within 
brings home the realities that in all ages there are conformist ideologies that must be reckoned with when innovative 
discoveries are made. This layering of related meanings is so complex that designing “in situ” , i.e. designing while 
playing the game or re-enacting a face-to-face scenario with actual students, is the only way that the designer will be 
able to uncover the subtle options to include or exclude challenges and context that the students will ultimately 
encounter. 

Designing in situ also requires that one begin along an iterative development path that requires the creation and 
testing of a series of increasingly authentic replications of the final environment (Schon, 1983). For instance, in the 
game described above, we have first written out a storyline where students have given us feedback as to their 
interest in solving problems for science in this context. Based on their feedback we made revisions and designed a 
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board game where game mechanics drove the play and encounters with the problems to be solved. We brought 
teachers, students, and fellow designers in to play this board game and received enough feedback to completely 
redesign the board and add the idea of a computer environment that is coupled to the board game play. During each 
of these iterations we examined the MDC to check off which chunks of content we were able to address with each 
version. The higher level learning skills and the levels of engagement desired drove our move to the more complex 
environment described in the outline. We still are only at the “mock-up” stage from a visual fidelity point-of-view, 
but we found that the hardest things to design are the game mechanics (see Figure 3). 




Figure 3 

Game Board (2 nd Iteration) 



Mechanics can only be designed by observing students playing a game and making choices within that context. 
Tweaking the mechanics, making content more or less available, creating more functionality within the game, are all 
elements we are focusing on at this stage and in subsequent iterations. Once we have the game-play running the 
way we want, then we will make it even more engaging via higher resolution graphics and animated cut-scenes on 
the computer. 

4. Define Learning Vectors that couple to an Experiential Mode Framework 
as well as linking game challenges to the student interactions. 

The Learning Vectors are defined as the actions a student engages in that match with the prioritized content chunks 
in the Master Design Chart. For instance the concept of "Weight" is pertinent in the comparison between 
Aristotelian concepts of weight as it affects a falling object, and Galilean teleology which is counter to Aristotle’s 
view. In a dynamic learning environment where students have many choices to pursue, any action may be 
considered a vector. Only those actions moving them closer to achieving a level of learning as stated in the MDC is 
considered a learning vector. During game -play analysis there are often actions identified that are not productive for 
the student’s learning, and when a perponderance of non-learning vectors are identified, this may signal the need for 
redesign of the mechanics, the LE, or a change in the storyline. A classic case of this may be found through 
observing the game -play of Oregon Trail (Company, 1986), where it was intended to offer functionality to gather 
food via a deer hunt, but students traditionally never make adequate progress toward reaching Oregon because they 
spend too much time shooting deer. Thus gathering food would be considered a learning vector, but slaughtering 
deer heards would be a non-learning vector. 

Once the desired level of learning vectors are observed in game -play, or in any other context desired, a script should 
be constructed that describes a successful path through the learning environment. This script may be in a word 
processing document consisting of a table with 3 columns and many many rows. The entire experience may be 
broken down into Scenes and these broken into Scenarios. The Scenarios describe all of the actions that take place 
within a scene, and there are many games where multiple scenarios may take place depending upon the choices of 
the students playing the game. This brings up the need for a script technique that handles branching, and I 

recommend the “jump to page ” technique preceded by a statement such as “ if a player selects X, go to page 

”, It is true that hard-copy scripts may end up being cumbersome, but this is a necessary evil if groups of people 
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are working together in the design (see Figure 4). To increase efficiency in design group collaboration during the 
last two steps, a web site that allows for postings by each of the development teams (most often the Instructional 
Design Team, the Story and Game/Play Team, the Engineering Team, and the Graphics/Sound Team) (Appelman, 
2008). Using a web resource such as this allows for any member to review the MDC, linkages to the mechanics, 
story, graphics, and programming functionality. For instance, for the iKIDS game it is envisioned that the computer 
addition to the board game will allow for the running of simulations on falling objects such that trial and error 
learning may be utilized to determine the correct challenge questions that is drawn from the “Physics”card stack. 
The development of this simulation module would need input from all the other teams to play and offer critique. 



opjXlMJICn SCMFT 

INFO Elements 



FORMULA 
o VARIABLES 
o Defining a 
CONTEXT for 
INQUIRY that 
matches the 
problem 
o BLENDING 
MATH and 
PHYSIC to solve a 
problem 



S/29/201 1 

INTERACTIONS 

PC could ask to know 
more about Aristotle's 
Philosophy that 
triggers either a short 
explanation from 
Galileo, the History 
NPC, or they decide to 
beam to Aristotle's 

Timeframe 

Looking at resources 
Turning pages 
Pointing at critical 
elements through 
cursor roll-overs 
where key elements 
highlight 

MAPs of ITALY that 
show location of 
PIZZA 



Experience 
SCENES IN 
ITALY 



Either move outside 
and use period travel 
or hop into their 
transporter 



What Galileo 
invented 
The state of science 
at that time period 
What did people 
eat back then 



Unlimited objects to 
create and interact 
with 

Consume period food 



BRANCHING : to cut scene of 
GALILEO'S explanation —OR — 
BRANCHING : to iKIDS 
HUDDLE and cut scene of 
HISTORY NPC ~OR~ 
BRANCHING : to ARISTOTLE - 
OR - 

They continue as follows . . . 

At this point our Physics NPC can 
look at one of the resources he 
brought and look up "WHEN 
THINGS DROP”. She comes 
across the formula for calculating 
the speed of a falling object that 
has distance, & mass times a 
constant. 

The Math NPC then says ohh| 
that ' s easy and says all we need is 
a known height of something and 
drop two different mass objects. 
Galileo says that he knows the 
perfect place for this ... the 
Leaning Tower of Pizza. One of 
the other NPC’ s says “It’ s made of 
PIZZA?” and GALILEO, after 
laughter stops, says it’s named 
after the town w e are in. 



There is a choice here to take time 
to travel to PIZZA via period 
methods, or to use their transporter 



There is also time while Galileo 
gathers the objects he w'ants to 
“drop” for the PC to look around 
Galileo’s lab and play with tools 
and objects there. 

Galileo might also offer them food 
and they may need to up their 
“health meters” at that point 
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INTERACTIONS 



o Structural 
Engineering 
Content 


o Navigating around 
tower 


The gt.Qup.gQ«mtbe,t.om.of 
PIZZA and navigate to the Tower. 
One of our NPC’ s asks why it is 
leaning and Galileo explains that 
they did not test the ground 
adequately under the structure 
because they thought it was stone. 
Galileo could also quip that he 
“told them so, but they would not 
listen to him”. 


o TOOLS THAT 
MEASURE 
DISTANCE 
o TOOLS THAT 
MEASURE 
ANGLES 


o Locating tools in 
backpack 


Our PC asks how tall it is, and 
Galileo says that he does not know 
that. At that point our PC says 
don’t worry, that’s what we are 
here for, and then looks in his 
backpack for two things . . . a 
pedometer and a sextant which are 
labeled “HOW FAR DO I WALK 
and “WHAT IS THE ANGLE”. 


o TRIGONOMETRY 
FORMULA FOR 
SIDES AND 
ANGIES OF 
TRIANGEL 


o Find F ormula in Math 
Reference 


The Math NPC mentions she has 
the formula for determining height 
and shows it to everyone 


o HOW A 
PEDOMETER 
WORKS 
o HOW A 
SEXTANT 
WORKS 


o ENTERING PACE 
INTO PEDOMETER 
o Navigating to base of 
Tower and then taking 
measured steps back 
to group 
o SIGHTING 
THROUGH 
SEXTANT and 
reading output 


Our PC walks to the base of the 
tower and sets the PEDOMETER 
to his STRIDE distance, and walks 
back to the group. He then asks 
his Math NPC to enter the result 
displayed on the PEDOMETER 
into the correct location in the 
formula. Then our PC takes the 
SEXTANT and sights up the tcp of 
the tower and reads the angle 
which is given to the Math NPC. 


o TRIGONOMETRY 
FUNCTIONS 
o FORMULA FOR 
HEIGHT 

o VARIABLES OF 
SIDES AND 
ANGIES 


o Entering data into the 
formula 

o Interactions with 
Galileo and group 


Everyone gathers around the 
formula and our PC clicks on 
calculate, which gives us the 
height of the tower. 

We now take a picture of the tower 
and the data is now displayed over 
the tower with distance and height. 
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Scenario Script 



The LE is complete when game -play includes adequate levels of learning vectors observed, and the engagement is 
such that the students want to leam more content in the same manner. It also is complete when teachers and school 
systems wish to include these types of learning environments. If the design is adequate, then national and state 
standards in the specific content areas will be met, and the same style of learning environment can be applied to 
other content. The connections in scientific inquiry for Aristotle, Galileo, and Newton relative to falling objects, can 
also be made for Leonardo DaVinci, and the Wright Brothers for air-foil physics. Not only does this strategy lend 
itself to develop LEs on a wide variety of topics, the 4xEID Process provides a development process that works 
equally well with any subject-matter. 



In this paper an attempt has been made to stress that if an instructional designer wishes to venture into experiential 
education (Dewey, 1938) with a constructivist epistemology (D. Jonassen, 1999) an adherence to a specialized 4- 
Step Experiential Instructional Design Process (4xEID) is advisable. This is because dealing with the levels of 
dynamic design challenges to appropriately offer adequate access to content resources in an engaging manner, the 
complexity requires both a structured content analysis and a design process that can be used in situ. The 4xEID 
process is designed for multiple in situ iterations and is not content specific. In other words it can be used to aid 
students learning how to interact with customers in a store environment, to learn strategies and techniques of playing 
soccer, to teach science knowledge, or to learn any other decision, or skill, based content. 
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Abstract 

The need for the sharable content object reference model (SCORM) to decrease the size of its shareable content object 
(SCO) is evident, especially since the introduction of Web 2.0 environments and new delivery systems. If SCORM is to be 
part of these emerging technologies, it needs to decrease its SCO size to the activity level to allow greater reusability, 
repurpose, adaptability, and portability of its learning objects. This will keep courses current at a lower cost, as well as 
enhance the transfer of knowledge; it will also help teach competencies, which will boost productivity. Greater reusability 
will help increase mental models. Emerging technologies will require that SCORM incorporate new standards for 
navigation, as new mobile learning environments communicate in shorter segments, requiring smaller SCOs and a 
different data model. 



Background 

Advanced Distributed Learning (ADL) developed a collection of specifications and standards known as the 
sharable content object reference model, or SCORM, as a way to standardize e-leaming within the defense industry. The 
need arose because each government contractor had its own system and guidelines, resulting in many inconsistencies. As a 
result, ADL is now in charge of publishing, governing, and updating SCORM specifications and standards. 

There have been several versions since SCORM’s inception in 1997. The latest version, SCORM 1.3, launched 
in 2004, and includes the ability to specify sequencing of activities that use content objects, and resolve ambiguities. This 
latest version also allows using and sharing information, regarding success status for multiple learning objectives or 
competencies across content objects and across courses for the same learner within the same learning management system 
(LMS). It also contains navigation abilities. The current version is 1.3.4 (SCORM, 4th edition). It was released in 2009 
and includes testing requirements and content packaging extension requirements. 

SCORM moves content, as well as students’ profiles and assessment records, from one platform or system to 
another, making data into modular objects that instructional designers and developers can reuse in other lessons. This 
enables any LMS to search other systems for usable content. SCORM does not ran the LMS, but allows interoperability 
between content and different learning management systems, regardless of the tool used to create the content. SCORM 
also facilitates navigation and provides content sequencing strategies. 

Introduction 

The introduction of the reusable learning objects (RLOs) inspired by the object-oriented programming practice in 
computers and the integration of SCORM have facilitated Web-based instruction, contributing to a major paradigm shift in 
education and training. This allows instructional designers and developers greater opportunities for creating new and rich 
virtual learning environments (VLEs) with reusability, adaptability, and repurpose in the way technology delivery systems 
present information to the learner. At the same time, the success of these interactions depends on its smallest unit, the 
shareable content object, or SCO. 

A SCO is the lowest level of granularity that communicates with the LMS. It has four requirements: it must be 
durable, reusable, accessible, and interoperable. A durable SCO is an electronic resource that does not need adjustments as 
learning technology develops over time. A reusable SCO is developed once; is context-independent and reused in different 
learning objectives or in other lessons. An accessible SCO is linked to a description of the content and can be found 
when required; an interoperable SCO is one that can be launched correctly by various VLEs, (Bailey, 2005). 
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SCORM and its Components 



The Function of RLOs 

The use of RLOs allows database-driven programs to store these objects only once, while using them many 
times throughout a course. Furthermore, it provides learners with a variety of visuals and updated text in their 
lessons. The use of RLOs allows VLEs to be highly interactive and adaptive to ongoing changes, keeping education 
and training current by making changes to only the content areas that need an update versus the entire course, which 
could be too expensive, thus saving space and money. 

It is also important to establish a common data model that will ensure that all SCO information can be 
tracked by the different LMSs or VLEs, since SCOs are designed to be shared with other LMSs and VLEs . At this 
point, it is important to establish the parameters of what will be in that model, to establish common ways for 
SCORM to report the information to the LMS. For example, if a test SCO uses a unique test scoring system, the 
LMS may not know how to interpret the test score or process the given data. However, using a common data model, 
the LMS will be able to interpret the data according to the parameters (ADL, 2009b). 

We all need to have the same definition and understanding of what an RLO is, because we are sharing these 
objects, sometimes from different places and maybe with different LMSs. Currently, there is no agreement on the 
type of RLO, or how small or granular it should be. Nor there is any agreement on the type of information or 
metadata one should include when describing the SCO or RLO (Churchill, 2007). For instance, some suggest that 
RLOs can only be of digital form (McGreal, 2004; Smith, 2004), while Wiley (2000) and the Institute of Electrical 
and Electronics Engineers (IEEE, 2002) suggest they are predominately instructional components, digital in nature, 
that support learning. Moreover, there are definitions that talk mostly about their function, comparing them to 
LEGO™ building blocks that can be put together in any number of ways to produce a desired learning outcome, 
because they are standardized with a uniform pin size so this task can be accomplished (Hodgins & Conner, 2000). 

Churchill proposes a classification containing six types of learning objects: presentation, practice, 
simulation, conceptual models, information, and contextual representation objects. He also proposes that it would 
integrate various media modalities (text, animation, tables, diagrams, video, etc.) into a single learning intervention. 

Even if all RLOs were digital, none of these definitions agree as to an RLO’s size and its granularity. Since 
a SCO interacts with diverse VLEs and LMSs, then this agreement is of the utmost importance. Without an 
agreement on SCO size and granularity, it will not be possible to have high SCO reusability, transferability, or 
interoperability between different proprietary models, which according to Hodgins and Conner could only happen 
with open accredited standards. 

SCO Size 

SCORM’s success will depend on the size of its SCO, a collection of one or more assets grouped together. 
It is the smallest object that communicates with the LMS; the SCO initiates all communication between the LMS 
and the SCO. Once the SCO is launched, it exchanges information with the LMS that can store and retrieve it. 

The size of a SCO is affected by branching decisions and the amount of information required for a given 
learning outcome. The lesson SCO size most used in companies is a one-to-one relationship with the terminal 
objective (TO), which does not allow instructional designers to create new paths for reusability within the same 
lesson in the form of enabling objectives (EOs). However, if SCOs were smaller and they had a one-to-one 
relationship other EOs, several EOs could be clustered to work toward the TO, allowing for the creation of new 
paths for reusability (Chapman, 2007). 

Currently, SCORM does not have a standardized size for its SCOs. Even though many government 
contractors have come up with their own SCORM-compliant software to develop their training products, each 
company determines the size of its SCO. However, ADL recommends that for greater reusability, SCOs should 
consist of small units. Therefore, to achieve a higher level of interaction, repurpose, reusability, and transferability 
using SCORM, the size of its SCOs need to decrease from a lesson to a lower level of instructional intervention. 
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Advantages of Smaller SCOs 



Many instructional interactions, when made into smaller SCOs, can be reused in various sections or parts of 
a section throughout the same lesson, such as an opening presentation, a later definition, as part of a remediation, or 
summary. Moreover, some learners may need to repeat some activities at different levels of interaction, in which 
case smaller SCOs will allow the learners to move with ease from place to place. The same information will help the 
learner connect to the content by relating RLOs of similar tasks (eliciting prior knowledge) as the learner interacts 
with new material. The larger the SCO, the harder it is to reuse and adapt to other content; conversely, the smaller 
the SCO, the greater its reusability, adaptability, transportability, and ease of navigation. 

Other benefits of smaller SCO size deal with cost benefit and productivity. Smaller SCO size will keep 
instructional programs current at a lower budget. This could solve budget problems for many institutions by 
updating only those SCOs that have become obsolete and thus keeping programs current at a lower cost. Similarly, 
with smaller SCOs, competencies can be taught as job-related tasks, increasing productivity. 

Metadata and SCO Size 

Metadata is “data about data,” (ADL, 2009a). Metadata is information that describes the SCO and its 
components; it is a form of labeling that enhances the search and discovery of these components. Metadata is used to 
facilitate discoverability and reuse of learning resources; it contains information such as content title, description, 
date of creation, and version. Each RLO has its own metadata as well, information about the object that serves as a 
cataloging information system (Smith, 2004). The Learning Object Metadata (LOM) (IEEE 1484.12.1-2002) is one 
standard used internationally that defines all of the data elements in terms of interrelationships that are both 
hierarchical and iterative. Those who use the LOM standard should follow its structure for maximum 
interoperability. 

Larger SCOs have less descriptive metadata than smaller SCOs. A larger shareable content object that 
includes an entire course is context-independent, which means it is not related to any content organization structure, 
whereas a smaller SCO allows the designer to include context such as activities, tasks, commonalities, or 
competencies. Words selected as metadata should be universal and standard (used by everyone in the profession) so 
they can be easily tracked by other SCOs, achieving interoperability between systems. Activity metadata, on the 
other hand, should not only be universal, but also contain context-sensitive information. It should describe the 
activity in relation to its purpose, who can use it, etc. (ADL, 2011; Duval, 2001). 

Reducing the SCO Size 

A way to reduce the size of a SCO is by dividing it into various RLOs by means of activities instead of 
lessons that contain numerous activities, most of them similar in nature. Many activities share some of the same 
tasks and/or instructional events. Another way to create a small SCO is to generate the SCO itself, by combining 
assets (e.g., text, images, sound) or RLOs related in content and from there assembling them into a SCO. 

Depending on its content, an activity may aggregate more than one SCO. For instance, when designing a 
lesson that involves engine maintenance, the designer would need to explain only once the importance of changing a 
filter or performing an oil change (no matter the type of engine). What changes from component to component are 
the periodicity (how often something is done) and the location of where it takes place. This example requires the 
SCO metadata to be more descriptive, for it relates to an activity that can be replicated in many places within the 
same lesson, as well as in other lessons. All those items that share the same content area or have a commonality are 
grouped into the same explanation, becoming a reusable SCO. Only the distinctive attributes or differences are 
placed into separate SCOs; thus increasing the number of SCOs per lesson by decreasing the SCO size according to 
shared tasks, activities, or commonalities. 

A smaller SCO becomes transferable to many other lessons, depending on its metadata, which describes 
the elements of the content package in its file. Metadata allows learning resources to be found by the LMS when 
stored in a content package or a repository. It is a best practice to describe learning objects with metadata; it 
increases their reusability by facilitating their search and discovery across LMSs. 
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SCOs and Data Models 



According to ADL (2009b), a data model is a standard set of data elements used to define the information 
being tracked for a SCO’s completion, such as a quiz, a test, or an input interaction. It is the information that both 
the LMS and SCO need to know for reusability across platforms. 

When designing an interactive instructional intervention, the activity metadata will support SCOs that will 
communicate with the LMS. The SCORM Run-Time Environment (RTE) deals with this communication. As the 
learner interacts with the content, the LMS evaluates the performance of the interaction and identifies an activity 
that corresponds with its input. The LMS next launches the corresponding SCO from within a VLE and presents it to 
the learner. It then tracks and stores the records of the learner’s activity along with the SCO. This is the function of 
the RTE (Bailey, 2005). 

For SCOs that will be shared with other VLEs and LMSs, it is critical to establish a data model that will 
ensure that the information will be tracked across systems. It is important to establish the parameters of what will be 
in that data model, creating common ways for SCORM to report the information to the LMS. These rules are also 
central for sequencing and navigation. 

Smaller SCOs will require a different data model, as there will be a greater level of interaction between the 
learner and the VLE. The greater the interaction the superior the educational experience for the learner. However, 
SCORM standards will have to adjust to these new challenges, as new technologies demand new data models. 

Reusability 

Increasing Reusability 

Dividing SCOs at the activity level allows greater reusability, transferability, and portability. A smaller, 
well-designed RLO can be reused in various ways: by different learners working on the same course, or by the same 
learner working on different tasks or activities, or solving different problems; it can also be used at different levels 
of knowledge or skill and in different disciplines that share the same competencies. 

To increase reusability, break large SCOs into smaller units, which can be used independently of each other 
but include as much contextual setting as required to support the content; make sure to change its metadata by 
providing information that is more descriptive and precise, relating the information to the activity instead of the 
lesson. This descriptive metadata will facilitate discoverability within a larger content repository. This way, each 
activity will have its own “activity” metadata repository through which a reusable learning object can be searched as 
opposed to a larger SCO that only has all its RLOs within a single metadata repository: the course or lesson 
metadata repository (see Figure 1). 



Current Lesson SCO Activity SCO Metadata 

Figure 1: Difference between the size of a lesson SCO and activities SCOs 
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This descriptive metadata will allow similar resources (i.e., tasks associated with the activity) to associate 
as RLO-relevant to the selected SCO. It will also allow the LMS to find resources based on similarities when 
necessary. 

To increase reusability, make sure each SCO is self-contained and can stand on its own. This means that 
aside from the connections to its own RLOs and the other SCOs, there are no links to outside material (Figure 2). If 
there are links to outside material, do not make its access mandatory for completion of the task within the learning 
object. Also, eliminate the RLO’s reliance on other learning objects that would prevent its mobility to other sharable 
content objects (Smith, 2004). 




Figure 2: SCOs dependent on internal and external links for completion of its task 

Reusability and Mental Models 

Reusability helps in the development of mental models, which are internal images of what is true and how 
the world works, created through computer human interaction. Some characteristics that will help build mental 
models are affordance, simplicity, familiarity, availability, flexibility, and feedback (Khella, 2002). For example, an 
activity SCO presented at various times becomes familiar to the learner; it later will help build upon prior 
knowledge. A frequently accessed activity SCO reinforces and enhances the mental model. Similarly, immediate 
feedback throughout the lesson helps in the development of the mental model. 

Similarities and Differences as Building Blocks 

SCOs can be grouped by commonalities or similarities in content, such as according to how components 
are made, placed, used, maintained, grouped, selected, and the like. In this case, a commonality may have several 
SCOs bound together by a function, purpose, location, etc. Whenever there are similarities, there are also some 
differences and contrasts that need to be explained and/or highlighted. Content aggregations by commonalities help 
in identifying competencies by recognizing job-related tasks used in Knowledge, Skills, and Abilities (KSA), which 
are required for a successful job performance in a job situation (Dubois & Rothwell, 2004). 

In the current lesson SCO size, as displayed in Figure 1, these commonalities and differences are part of an 
entire lesson (a large SCO); common activities cannot be separated and reused. However, if these sections were of 
smaller SCOs (i.e., activity SCOs) then they could have multiple functions by being separated and reused multiple 
times as different activities, as presentation, examples, and non-examples in other parts of the existing lesson and/or 
other courses. Likewise, commonalities themselves, when made into activity SCOs, can be reused in other parts of 
the lesson or other courses that share the same activities and/or have the same competencies required for a particular 
job. Competencies are important in training as they increase productivity by distinguishing what is necessary for 
exemplary performance (Rothwell, 2001). Instructional designers can also use commonalities as building blocks for 
the next section that shares the same tasks or competencies, thus increasing knowledge transfer. 

By dividing SCOs into smaller units, instructional designers can have a wide variety of uses for the same 
activities or RLOs. For instance, each SCO will have its own navigation system as each will have a subset of 
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reusable learning objects and metadata associated with it. Reusable learning objects become consumer-friendly, as 
they can be reused multiple times not only as a parent activity but as subsections of parent activities as well, 
according to their commonalities or differences. Amount of reusability per lesson will depend on the instructional 
interaction and instructional strategies, which provide a greater learning experience as the transfer of knowledge is 
increased. Figure 3 depicts a SCO that is presented to the learner and later recalled several times as it is compared to 
new material. 




Figure 3. SCO Reusability within the same lesson. 

SCORM and Emerging Technologies 
Emerging Technologies, Reusability, and Transportability 

Many new delivery systems, such as those use in wdeaming (mobile learning), transfer only small sections 
of lessons at a time and/or activities that can be transported on small video segments or selected text (e.g., via iPod, 
podcasts, mp3 players, cell phones, or other applications). If SCORM is to be part of the new and evolving 
communication technologies, it will have to reduce its SCO size and modify its communication system to allow for 
content aggregation at a lower level. For instance, instead of aggregating all content under one objective, it would 
have to aggregate small amounts of content under various activities, each activity linked to an enabling objective 
(EO), which is part of a terminal objective (TO). This allows the presentation of each activity as a separate unit, 
under the umbrella of the same EO. The LMS will present the activity SCO to the learner; then will evaluate the 
learner’s interaction according to the input received. From there, the LMS will launch the corresponding activity 
SCO, according the TO’s sequence, and present it to the learner until all EOs have been fulfilled. In this way, each 
activity can be organized separately; some can be reused later as needed (e.g., as presentations, examples, non- 
examples, feedback, and/or remediation). Based on the descriptive metadata as well as the navigation and RTE data 
model, the LMS will have to determine which SCORM content needs to be delivered next. 

As for navigation, SCORM sequencing does not put restrictions on how EOs and TOs are associated with 
activities; nor does it make any assumptions on how to interpret learning objectives (ADL, 2009c). It is therefore up 
to the instructional designer and developer to divide SCOs into smaller units and allow selections by categories 
and/or specific activities to achieve greater reusability, adaptability, and transferability of content. 
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SCORM and Web 2.0 Environments 



Web 2.0 technology systems involve multiple authors, while SCORM was created for a single author or 
team. Although the latest SCORM version allows input for learner interaction, it is mainly in the form of 
assessment, since each question is considered an interaction. At this point, SCORM also lacks a model with 
standards that would allow interaction between the instructor and the learners, as well as and between learners 
themselves. These, among other constraints, make SCORM unsuitable for Web 2.0 environments (Rogers et ah, 
2007). 



As Web 2.0 technology and new delivery systems become available, such as those in use in /cleaning (e.g., 
podcasts, mp3 player, iPod, and iPad), new mobile applications and virtual spaces that communicate in shorter 
segments will demand higher reusability, transferability, and portability, requiring smaller SCO size than the current 
lesson size. If SCORM is to be part of these new tools, it will need to decrease its SCO size to the activity level. 
Emerging technologies will also require that SCORM have new standards for navigation, as these VLEs behave 
different not only requiring smaller SCOs but also calling for a different data model. 

Summary 

Decreasing SCO size by activity content will allow instructional designers and developers to create 
interactive instruction, providing greater learning experiences and promoting a greater level of interaction between 
topics of the same lesson as well as between lessons of the same course, thus increasing transfer of knowledge. It 
helps provide feedback as well as remediation. Smaller SCO sizes will also help keep programs current by updating 
only those SCOs that have become obsolete, keeping programs current at a lower budget. Similarly, competencies 
can be taught as job-related tasks, increasing productivity. Learners, on the other hand, will be able to connect 
learning experiences, such as tasks and processes, in a meaningful way without having to relearn steps; they will be 
able to engage fully in interactive lessons. 

If SCORM is to survive the future, it will have to evolve to the point that it will integrate the Web 2.0 
technology, since these applications and spaces, such as blog technology that allows more participation and social 
interaction among learners, are already being incorporated into the curriculum. Furthermore, educators are using 
multiple formats across several platforms to increase the chances that their message will reach the intended 
audience. Education and training is communication; while the basic message is not changing, the vehicle is. . .and 
fast. 
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Abstract 

The purpose of this paper is to discuss the authors’ experiences with pedagogical Skyping. The authors 
strived to engage classroom interactions between America, Africa, Europe, and Asia. In the past year, the authors 
successfully and actively engaged learners synchronously via Skype for a refugee college in Thailand. The paper 
will present findings of student performance based on their course grades and teacher performances based on student 
evaluations. The authors introduce a new term edSkype 3.0. edSkype 3.0 includes teaching, engaging, and delivery 
via Skype; using sound instructional design strategies; collaborating video and audio chats, texting, video 
conferences and webinars; integrating pedagogical YouTube videos, iTunes, pod casts and screencasts; and 
conducting synchronous and real-time distance education with authentic audiences via Skype technology. The 
participants for the study included a total of 32 students enrolled in the Anatomy and Physiology class and the 
World Geography. There were a total of 19 (six females, thirteen males) students in Section A and 13 (seven female, 
six male) students in Section B class. The finding indicated that Online Skype learning can also be as beneficial as 
the On Campus learning. The findings provide preliminary insight into further research to find similarities and 
differences in the two learning environments. 



Introduction 

Internet access in schools and educational environments has allowed interactions between different types of 
participants all over the world. Skype technology has made this form of communication easy and affordable. As 
such the slogan for Skype is “the whole world can talk for free” (Fryer, 2006). Skype is a web tool that helps convert 
a computer or phone into a web phone. It uses Voice-over-Internet-Protocol (VoIP) software to promote conference 
calls, video and audio chats, texting, and file and screen sharing. Hines (2005) predicted that these sophisticated 
features would serve as a driving factor for the future of technology and increase its popularity in classroom 
environments. 

edSkype 3.0 basically means creating and engaging virtual classrooms via Skype technology. It involves 
collaborating video conferences and webinars; integrating YouTube videos, iTunes, pod casts and screencasts, and 
virtual presentation and synchronous and real-time distance education with authentic audiences. 

edSkype 3.0 Relevance And Practices 

In recent years edSkype has become very popular. Skype has been used as an easy tool to integrate 
classrooms worldwide. It allows peer-to-peer learning by connecting students all over the world to help expand their 
knowledge and skills. Students are now more involved and engaged in learning. Peer-to-peer communications are 
becoming more popular with participants being able to communicate with authentic audiences rather than being 
limited to mere textbook education. This technique has benefited both the teachers as well as parents (Nelson, n.d.; 
Rosen, 2009). Subsequently, various cultural languages are being taught using this technology (Kilian, 2010). As a 
result, unfortunately, many companies are trying to make money. 
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Skype users in Thailand or India can call a Skype user in England or United States. Currently, schools in 
Waltham, Massachusetts are forming Skype pen pals to build virtual communities. For example, where participants 
would have missed school due to sickness, they now have the opportunity to stay connected to the class via Skype 
(Moulton, 2009). Educators are currently trying to come up with new and innovative ideas of using Skype. For 
example third graders at Northeast Elementary connected with sixth graders at Kennedy Middle School graders to 
collaborate peer-to-peer research projects and creating virtual presentations (Rosen, 2009). In the blog posting by 
Moulton (2009), teachers from Michigan shared experiences of connecting with classes in, Italy, North Carolina, 
and Mexico. A teacher from Finland shared experiences of connecting with Europe and Asia. Another teacher 
described experiences in teaching history by collaborating with individuals residing in historic locations that are 
mentioned in the textbooks. Often, too, teachers partner with textbook authors to conduct classroom webinars to 
collaborate conversations between the author and the students allowing easy access of communicating directly with 
experts in the field (Moulton, 2009; Rosen, 2009). Eventually proposals were made to organize a Skype phonebook 
for educators interested in connecting with other educators (Moulton, 2009), as such, today Skype integrated the 
digital directory database called “Skype in the classroom.” This database allows interested educators to form 
connections to help collaborate classroom webinars with content experts, educational leaders, textbook authors, 
schools, colleges, and universities all over the world. 

Locating Skype Collaborative Classrooms 

The biggest challenge today is to find educators who are willing to facilitate classrooms via Skype. Many 
educators have used Merlot to find guest speakers who are experts in their field, via the Merlot member directory. 
This technology added a new dimension to the educational experience for participants (Merlot, 2010). Another 
reputable Internet resource for connecting and collaborating projects with learners worldwide is ePals Global 
Community ®. ePals is the world’s largest network of K-12 classrooms locally, nationally or internationally. The 
site offers discussion forums and encourages safe authentic exchanges with more than 200 countries; it allows 
communication and collaboration of activities from school and home. The ePal Virtual Learning Space helps create, 
share, manage, and collaborate educational content. The ePals Learning Space offer safe web 2.0 communications: it 
includes email, blogs, wikis, forums; share files like Word, PowerPoint, PDFs, pictures, audio/video; and conference 
with guest lecturers and experts (ePals Global Community, 2010). Many teachers have shared experiences of finding 
mutual classrooms via ePals as phonebook resource. More recently educators have turned to Skype. Companies, 
however, are moving more towards advanced applications such as Adobe Connect, WebEx, and GoToMeeting 
(Kilian, 2010). 

Recently several experts have shown a willingness to connect their classroom via Skype. A web service 
called “Skype for Educators” has worked on constructing a Skype phonebook database to aid communications 
between classrooms. The site provides names, e-mail, and expertise of individuals willing to help enhance your 
classroom engagements (PBWorks, 2010; Skype for Educators, 2009). 

Refugees Embrace edSkype 3.0 

Burma, for years, has been trying to overthrow a brutal military government. The military junta has 
enforced fear for the past two decades by killing 3000 people and holding more than a 1000 political prisoners 
including Aung San Suu Kyi. Runaway Burmese migrants constantly run to Thailand for shelter in refugee camps, 
on several occasions many are detained and sent back. Thailand supports refugees with organizations like “Migrant 
Educational Coordination Center” to provide basic education to self-sustain. As a result of the stand taken by the 
Thai government, the number of participants in these schools has grown tremendously over the past years. Today, 
there are almost 40% college graduates and about 50% high school graduates (Raj, 2010a). 

Kaw Tha Blay Learning Center (KTBLC) is a migrant school located on the Thai-Burma border under the 
sponsorship of Project Umbrella Burma. It moved from Karen State in Eastern Burma due to conflict caused by 
Burmese Military Regime to the current location in Tha Song yang, Thailand. The students physically helped build 
the schools infrastructure. This school facilitates migrant Burmese students from neighboring country of Burma. The 
main mission of the school is to equip stateless communities with basic knowledge and skills including cultural and 
language exchange. The Center teaches critical thinking, leadership skills, community health, computer training, 
spoken English and Thai, accounting, geography, teacher education, and human rights. Today it is officially listed 
under the Global Positioning System (GPS) map set up by the Thai Ministry of Education (Raj, 2010b). 
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KTBLC has long depended on foreign volunteers for the success of their teaching and learning. Due to 
KTBLCs dependency on foreign volunteers it ultimately became very difficult to get teachers physically to the 
school. Obtaining visas and documents; anticipating volunteer availability and flexibility; and dealing with lodging, 
travel, and food expenses are some of the hardest issues. 

edSkype proved to be the most Adaptable Educational Technology for Refugee (AETR). For instance, at 
KTBLC the free and user friendly Skype application proved to be the most successful and time saving tool 
compared to expensive and complicated web tools such as Tandberg ®, Elluminate ®, and Blackboard ®. 
Additionally, it tremendously cut costs to maintain volunteers. 



Participants 



Total Faculty 

In the spring of 2010 KTBLC recruited two Online Skype Faculty (OSF) to join the team of five On 
Campus Faculty (OCF), this team included a mix of three local faculty and two foreign volunteers (see Figure 1). 
The two new online recruits taught Anatomy and Physiology, and World Geography. On Campus Faculty taught 
community health, English language, computers, Thai language, technical cards, clinical guidelines, basic nursing, 
medicine, and social studies. As such only overall comparisons were made on student scores for online versus on 
campus courses. We were unable to make parallel comparisons between synchronous online courses versus 
synchronous on campus courses. 
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On Campus Faculty Online Skype Faculty 



Figure 1: Demographics of OCF and OSF 
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Total Participants 

The total number of participants enrolled in the online class was 32 participants. Each course included two 
sections, Section A and B. There were 19 participants (six females, thirteen males) in Section A and 13 participants 
(seven female, six male) in section B. The participants for this study consisted of individuals that participated in 
Skype classes for the spring 2010 semester (see Figure 2). 
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Figure 2: Participants Demographics: Females (13) and Males (19) 

Methods 



The data was collected for two performances: faculty performance and student performance. Faculty 
performances were based on student ratings conducted via end of the year surveys. The questions included content 
clarity, content organization, content expertise, student motivation, student participation, assignments, assessments, 
multimedia, and other outside support. Student performances were based on their achievements on their final grade 
scores. The grading criterion for World Geography and Anatomy and Physiology course were: Grade A = 81 to 
100%, Grade B = 71 to 80%, Grade C = 61 to 70%, Grade D = 50 to 60%, and Grade F = below 49%. The two 
major contributing factors to student scores were midterm and final grades. The types of questions included: till in 
the blank, true or false, multiple choice questions, matching, labeling parts, and short answer questions. 

Total Hours for World Geography and Anatomy and Physiology Course 

World Geography course was a four hour course (see Figure 3) taught twice a week while Anatomy and 
Physiology was a three hour course (see Figure 3) taught three times a week. World Geography was taught for two 
hours every Tuesday and Thursday from 9 pm to 1 1 pm United States CST (that is, Wednesday and Friday mornings 
from 9 am to 1 1 am Thailand time). The total number of hours for Anatomy and Physiology was three hours which 
was taught for an hour every Sunday, Tuesday, and Thursday from 8 pm to9 pm United States CST (that is, 

Monday, Wednesday, and Friday from 8 am to 9 am Thailand time). 




World Geography Anatomy & Physiology 



Figure 3: Total number of hours taught online per week for World Geography and Anatomy and Physiology course 
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Findings 



Between Sections Disparity Among The Courses 

All the participants received a Pass Grade in both the sections of World Geography course, however, 8% of 
participants failed in section B of Anatomy and Physiology course. About 50% of the class scored above average in 
both the courses. In the Anatomy and Physiology course 58% of the participants had an above average grade in 
section A with 54% for section B participants (see Figure 4) indicating a 4% disparity between the two sections. 
Section B of World Geography course performed extremely well with 85% of participants scoring above average 
with 43% participants in section A (see Figure 4) indicating a 42% disparity between the two sections. 
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Figure 4: Grade Disparity between the Sections 

Gender Disparity within World Geography Course 

Overall, section B outperformed section A in the World Geography Course. The male participants in 
section A outperformed the female participants by 37% (see Figure 5) while the female participants in section B 
outperformed the male participants by 3% (see Figure 5). 
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Figure 5: Gender Disparity in Scores for World Geography Course 
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Gender Disparity within Anatomy and Physiology Course 



Over all section A outperformed section B in the Anatomy and Physiology Course. Once again the male 
participants in section A outperformed the female participants by 12% (see Figure 6) while the female participants in 
section B outperformed the male participants by 8% (see Figure 6). 
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Figure 6: Gender Disparity in Scores for Anatomy and Physiology Course 



Faculty and Student Performances 

Although the OCF teaching performance was rated superior by the students with 26% higher ratings 
compared to the OSF (see Figure 7). The test scores indicated student performances scores high by 6% in the Online 
Skype environments compared to the On Campus environments. The average performance of students in the On 
Campus environment was 70% compared to 76% in the Online Skype environment. Thus, the average performance 
of student slightly exceeded in the Online Skype class by 6% compared to the On Campus class (see Figure 7). 
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Figure 7: Overall Faculty and Student Performances in Online and On campus courses. 



332 





Discussion 



The finding indicated that Online Skype learning can also be as beneficial as the On Campus learning. As 
such, Skype can be successfully used as an alternative technique for the purpose of teaching and learning. The 
success of this tool, however, depends on the effective use of sound instructional design strategies, along with the 
integration of appropriate pedagogical videos (YouTube) and audio chats (iTunes, podcasts), texting, screencasts, 
video conferences and webinars to help deliver synchronous and real-time distance education with authentic 
audiences via Skype technology. The findings provide preliminary insight into further research to be conducted to 
find similarities and differences in the two learning environments. 
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Abstract 

The usability of a statewide curriculum website was evaluated involving 876 participants. Site usage differed based 
on user group and overall site content and relevance was found to be good along with high levels of satisfaction with 
graphic design. Usability problems with ease-of-use, however, were identified leading to the suggestion that many 
potential users do not use the site because of difficulty finding the information they are looking for. Implications of 
the research center on the articulation of a usability evaluation framework that could be generalized for testing the 
usability of other websites as well as redesign considerations for the site examined for the study. 

Introduction 

The world of a teacher or school librarian is busy, bustling, and ever diverse. Time is of the essence and the 
demands on their time are constant. A state department of public instruction wanted to know what the state’s 
teachers, students, and school librarians thought of its curriculum resource center. Were they even aware of it? If so, 
how did they use it and how usable was it for them? A university research team was tasked with answering these 
questions. According to Nielsen and Loranger a website only has between 25-35 seconds to convince a user that 
they will be able to find what they are looking for and stay on the site (Nielsen & Loranger, 2006). 

Information Seeking and Web Usability 

Users searching for information are restless with a need for attaining the information that they seek. According to 
Taylor (1968), information seekers have a “vague sense of dissatisfaction’ (as cited in Case, 2007, p. 72) that causes 
them to seek out information that will help alleviate this need. Belkin, Oddy and Brooks (1982) also refer to a user’s 
information need as a “. . . recognized anomaly in the user’s state of knowledge” and Derwin describes the quest for 
information as “sense-making” (Case, 2007; Chow & Bucknall, 2011). Pirolli and Card (1999) refers to the 
information need as information foraging where humans are informavores (Denis, 1991, as cited in Pirolli & Card, 
1999) that hunt for information. Well designed websites high in usability therefore need to be both high in 
relevance and ease-of-use (Nielsen, 2005) and possess information architectures that have clear and strong 
‘information scents’ so users can satisfy their hunger for information and find what they are looking for as 
efficiently and effectively as possible (Chow & Bucknall, 2011; Morville & Rosenfield, 2008). 

One of the most commonly cited definitions of usability is the International Standards Organization’s standard 9241- 
11, which defines usability as the “extent to which the product can be used by specified users to achieve specified 
goals with effectiveness, efficiency and satisfaction in a specified context of use” (11) (ISO 9241-11, 1998; Jordan, 
1998; Alshamari & Mayhew, 2009; Alonso-Rios, Vazquez-Garcia, Mosqueira-Rey, More-Bonillo, 2010). 
Effectiveness is the “percentage of goals achieved, percentage of users successfully completing tasks and average 
accuracy of completed tasks,” efficiency is the “time to complete a task, tasks completed per unit time and monetary 
cost of performing the task,” and satisfaction is a “rating scale for satisfaction, frequency of discretionary use and 
frequency of complaints” (ISO 9241-11 as cited in Alonso-Rios, et. ah, 2010, p. 54). Jordan (1998) further defined 
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subcategories for effectiveness - which is comprised of task completion and quality of output - and efficiency - 
which is defined by error rate, time-on-task, deviations from the critical path, and mental effort. 

There are also significant differences between needs and preferences of adult and youth information seekers. 
Research has found that children's information-seeking behavior is characterized typically by browsing rather than 
searching for information using keywords (Large, Beheshti, Clement, Tabatabae, & Yin Tarn, 2009). A typical high 
school aged information seeker (14-18 years old) tends to have a short attention span, gets easily bored, scans rather 
than reads material (Fidel et al, 1999), and prefers larger font sizes (DiMichele, 2007; Nielsen, 2005). Pre- 
adolescent information seekers similarly prefer visual cues to just text and in general do not like to scroll. 
Information seeking emphasizes more exploration than hunting for specific information. The typical middle school 
student likes bright and engaging colors (Large, Beheshti, & Rahman, 2002) and sites that include animation, sound 
effects (Neilson, 2005), and bright colors throughout the background and the foreground of the site. 

The emotional or affective state in particular of youth information seeking has also been identified as an important 
factor (Dubroy, 2010) as negative reactions to a site or high levels of uncertainty may cause children to more likely 
abandon their information search altogether (Kuhlthau, 1991). In contrast, feelings of excitement, happiness, and 
enjoyment have been found to increase overall information seeking persistence (Bilal, 2005). The field of human 
computer interaction (HCI) has embraced the affective state as usability and cognitive processes have become more 
salient variables to consider in developing more highly usable websites and information spaces. According to 
Norman (2002), “A human being’s affective system is judgmental, assigning positive or negative valence to the 
environment”; immediate emotional reactions to interfaces, emphasizing “engagement, pleasure, and delight rather 
than just functionality” (Deng & Poole, 2010, p. 712) . Emotional reactions to websites in fact, may be a more 
significant factor to information spaces users tend to use than task efficiency (Bucy, 2000 as cited in Deng & Poole, 
2010). Schwarz labeled a user’s general affective reaction as “affect as information”, which represents a, “halo 
effect of emotional response toward an object carries over to the evaluation of object characteristics and general 
attitude to the object” (Schwarz 1986 as cited in Deng & Poole, 2010, p.712). 

In response to the growing research on usability in the literature, usability design standards have been developed that 
guide both preliminary design as well as serve as a checklist for existing sites. Ten usability design standards have 
been identified in particular as taking precedent (Jordan, 1998; Nielsen, 2005) with compatibility’ - designing a 
product compatible with user expectations - and prioritization of functionality and information - the most important 
functionality and information are easily accessible to the user - identified as the two most important usability design 
criteria to be aware of. 



Table 1 - Usability Design Standards 

Usability Heuristics (compiled from Jordan, 1998; Nielsen, 2005) 



1 . Consistency: Designing a product so that similar tasks are done in similar ways. 

2. Compatibility: Designing a product so that its method of operation is compatible with users’ 
expectations based on their knowledge of other types of products and the “outside world.” 

3. Consideration of User Resources: Designing a product so that its method of operation takes into 
account the demands placed on the users’ resources during interaction. 

4. User Control: Designing a product so that the extent to which the user has control over the actions 
taken by the product and the state that the product is in is maximized. 

5. Visual Clarity: Designing a product so that information displayed can be read quickly and easily 
without causing confusion. 

6. Prioritization of functionality and information: Designing a product so that the most important 
functionality and information are easily accessible to the user. 

7. Explicitness: Designing a product so that cues are given as to its functionality and method of 
operation. 
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8. Match between system and real world: The system should speak the users' language, with words, 
phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world 
conventions, making information appear in a natural and logical order 

9. Recognition rather than recall: Make objects, actions, and options visible. The user should not 
have to remember information from one part of the dialogue to another. Instructions for use of the 
system should be visible or easily retrievable whenever appropriate. 

10. Aesthetic and minimalist design: Dialogues should not contain information which is irrelevant or 
rarely needed. Every extra unit of information in a dialogue competes with the relevant units of 
information and diminishes their relative visibility. 



The purpose of the study is to determine the overall impact of a statewide curriculum resource center website 
utilizing a usability framework. Three research questions guide the study: 

RQi: Is the website relevant and easy-to-use for users? 

RQ 2 : How does the website rate in terms of efficiency, effectiveness, and satisfaction? 

RQ3 : How does the website compare to recommended web usability design standards? 

Method 

The concepts of effectiveness, efficiency, and satisfaction are typically experienced differently based on user status 
and perspective. Therefore, in order to ascertain overall usability, both the site’s users and the overall relevance and 
ease-of-use in which they can seek and find information must be determined. To understand the authentic 
perspectives of the divers set of users the site is designed to serve school librarians, teachers, and students at the 
elementary, middle, and high school level were asked to participate. The study’s collected data from 876 participants 
utilizing a statewide online survey, six focus groups (three teacher and three student), and 21 usability tests of 
teachers, students, and school librarians. 

Online Satisfaction Survey (n=816) 

A 13 -item online survey was administered over a two week period to school librarians and teachers send via email 
list serves as well as a link placed on the DPI home page. Overall, 816 participants completed the survey. 
Demographics of the respondents were not diverse as the sample was comprised of 90% female and 90% white. In 
terms of user group representation 50% were school librarians, 30% teachers, 12% students, and 8% parents. 
Elementary school participants represented the majority (41%) followed by middle school (31%) and high school 
(29%). The majority of respondents reported having access to broadband connectivity, 74% (n=602), while 20% 
reported still using a dial-up connection at home. In terms of prior experience with the site, 57% (n — 466 ) reported 
being expert users (visited 20+ times) and 25% reported being at least familiar (visited 5-19 times) with the site 
(n=200). 

School Site Visits (n=60) 

Sixty teachers, students, and school library media specialists also took part in six focus groups and 21 usability tests. 
Three different on-site visits to schools were conducted in different regions of the state: One elementary school 
(n=22), one middle school (n=12), and one high school (n=23). Usability testing with school librarians was also 
conducted at a local state conference (n=3). All were samples of convenience with the teachers and students being 
selected by the school’s media specialist and oftentimes under the auspices of availability. The participants at the 
conference were volunteers who agreed to participate in the study. 

Focus Groups 

At each of the three site visits a focus group of teachers and students was conducted using a series of 13 questions 
designed to collect their main impressions of the site in terms of usability - relevance, ease-of-use, and overall 
satisfaction - while walking them through the site using an LCD projector. 
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Usability Tests 

At each site six usability tests took place, three with teachers and three with students. Only three tests were 
conducted with media specialists. 

Usability test procedures. Each participant was asked to sign a consent form (18 years or older) or an 
assent form (under 18) along with a demographic survey. A brief introduction was read explaining the testing 
process and then the test began. The sessions were video-taped and timed using the usability software Morae. After 
completing five tasks deemed to be representative of major tasks a typical user might attempt to complete, 
participants were then debriefed using an instrument that measures their overall thoughts on the site’s effectiveness, 
efficiency, and their overall satisfaction. 

Usability Tasks. Teachers, students, and media specialists were asked to complete five representative tasks 
using think aloud protocol (TAPs) which is a protocol where the tester “thinks aloud” by verbalizing her/his thought 
processes as they attempted to complete each task. Examples of tasks used include for teachers, find reflective 
learning activities and icebreakers, for students, where would you go to learn how to properly cite a source ?, and 
for school librarians, find grade level competencies for computer skills. (See Appendix A). 

EES Post-Evaluation Form. Based on ISO Standard 924 1 definition of usability - how effective, 
efficient, and satisfying a product or service is - each respondent was interviewed and debriefed about their thoughts 
after each test. Utilizing Jordan’s more specific breakdown, each participant was asked to rate each usability factor 
on a scale of 1-10 (l=lowest, 1 (Miighcst). See Appendix B. 

Usability Heuristic Evaluation. One final tool used to evaluate the overall usability of the site 
was each member of our three member team conducted a usability heuristic evaluation which rated how well the site 
compared to 10 of the most prominent usability design heuristics or standards including consistency, designing a 
product so that similar tasks are done in similar ways; compatibility, designing a product so that its method of 
operation is compatible with users’ expectations based on their knowledge of other types of products and the 
“outside world”; and, consideration of user resources, designing a product so that its method of operation takes into 
account the demands placed on the users’ resources during interaction. The ten standards and instrument used are 
attached as Appendix C. 



Survey Results 

Survey respondents used the site mainly for research (27%), lesson plans (23%), professional development (17%), 
or school work (16%). 



Results 



Website Usage 




Homework-Own 

1% 



17 % 



Figure 1 - Website Usage 
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The most visited areas of the site involved references source (n=517), the professional zone (508), and parent zone 
(400). In terms of age groups served, the elementary school zone (n=331) and middle school zones (n=298) were 
much more frequently used then the high school zone (n=128). Overall, the site received a cumulative satisfaction 
rating of 8.1, which suggest that most participants were satisfied with the site. 



Table 2 - Most Visited Areas of the Site 

Most Visited Areas of the Site (f) 



Reference sources 


517 


Prof. Zone 


508 


Parent Zone 


400 


IMPACT 


382 


Elem. Zone 


331 


Middle Zone 


298 


Kaleidoscope 


279 


eBistro 


214 


SAS in School 


173 


Media/Tech Zone 


168 


High School Zone 


128 


Other 


83 


Hovercraft 


79 



Focus Group Results 

Teachers. Teachers associated the site for information about workshops (which many have previously used 
the site for), research, and lesson plans. When looking at the site’s home page it was unclear to them at-a-glance 
what its purpose was and what it could offer them. In terms of strengths and weaknesses of the site, teachers felt that 
the site was “student friendly” and relatively well organized. In addition, a number of teachers mentioned that they 
liked the simplistic yet elegant graphic design. They noted, however, that some of the language is not age- 
appropriate (i.e. elementary students are not going to know what a “citation” is in the elementary school zone) or 
well organized as they recommended breaking up the zones into grade levels when appropriate. 

In terms of overall purpose and reasons for using the site, use of the citation maker and access to online databases 
were most commonly mentioned and it was noted that the use of the site appeared to be much more prevalent on the 
elementary and middle school levels in contrast to the high school level. Many noted that they did not get a sense 
that people were using it and a number of teachers confirmed their own lack of use derived from not being aware of 
what is available to them and are unable to understand what there is “at-a-glance” and therefore they just use other 
sites. Other barriers toward using the site was it was “too professional” in appearance and did not really speak to 
either teachers or students. So for example, instead of having “zones” the main channel labels could be changed to 
“students” or “students and teachers,” which would go a long way in telling them what the site is about without the 
need to have attended a workshop. 

In general, teachers felt that the information and content the side provided was good but too distributed and not 
organized enough for them to understand what was available and remain on the site for very long. Time is extremely 
precious for them and if they cannot find it quickly they will move on to another resource. As one teacher put it: 

“I don’t know who would use this. It’s too much work to find things. We’re so curriculum driven that you’ve 
got to have things laid out with that in mind. As a teacher, we have so little time we need to see BAM let’s hit it 
let’s go there. Too much searching, we’re not going back there. When you’re searching you click on the top 
three and never make it past page two or three EVER” (teacher focus group response). 
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Students. For the most part, students were not very familiar with the curriculum site but their overall opinions 
were similar to the teachers in terms of satisfaction with the overall graphic design and use of colors but having 
difficulty finding information in an efficient, effective manner. Students reported having used it for class research in 
social studies as directed by teachers in-class. Overall, students were unable to clearly identify the site’s purpose 
from the home page. Several students thought it may be a school locator organized by grade level. In terms of 
general strengths students felt it appeared very organized with a nice graphic design. They could not really identify 
weaknesses as most had never used it or knew anyone that had used it. 

One of the primary aspects of the site students really liked was its graphic design - both in terms of graphics (the 
owl and clouds are a big hit) and color in terms of simplicity and tasteful use of blue and white space. Some felt that 
the lack of use of color was a strength and “not like all the other Web sites designed for kids” with “crazy” colors 
everywhere while others said they wanted more color and animation like they are used to on other kids-oriented 
sites. 

Usability Test Results 

Teacher Results. Overall, teachers were able to complete Task 3 {Find grant writing tips and list of grant 
opportunities ) and Task 4 {find reflective learning activities and icebreakers ) successfully 89% of the time. Task 1 
(find information about different student learner profiles) 78% of the time. Task 2 (Return back to the home page) 
only 33% of the time, and Task 5 (find a lesson plan) was not completed successfully by any participants. Also, as 
would be expected, the two tasks where teachers had the least success also had higher error rates and took longer 
than the other tasks. 



Table 3 - Teacher Usability Tests 

Teacher Task Completion 



Task 


Error 


Time 


Success? 


1 


1.5 


41.6 


78% 


2 


2.8 


55.0 


33% 


3 


2.4 


49.8 


89% 


4 


2.1 


43.9 


89% 


5 


4.0 


77.0 


0% 



When examining usability ratings the site’s cumulative rating across all teachers was a 6.0 at of 10. The site’s 
highest rating was a 7.0 for quality of output and it received its lowest ratings for task completion, error rate, and 
mental load. In terms of the usability factors, overall satisfaction (6.2) and effectiveness (6.2) were the highest rated 
while efficiency (5.7) was the lowest rated. 
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Teacher Usability Ratings 




Student results. Task 2 ( Return to the home page) was completed the most successfully at 75% while Task 3 
(Look up an encyclopedia article about Australia) and Task 4 (Find the first governor of North Carolina) were 
completed 63% of the time. Task 1 (Where would you go to learn how to properly cite a source ?) was completed 
only 38% of the time and Task 5 (Find information on internet safety) was only completed by 13% of students. 

Table 4 - Student Usability Testing 

Student Task Completion 



Task 


Time 


Error 


Success? 


1 


5.1 


117.3 


38% 


2 


2.6 


67.6 


75% 


3 


2.0 


37.3 


63% 


4 


2.5 


39.3 


63% 


5 


6.5 


125.0 


13% 



Student usability ratings conducted after the usability tests suggest that they liked the site graphics (8.8 out of 10) 
and were highly satisfied (8.8 out of 10) with the overall site. Error rate (5.8) and mental workload (7) were the 
lowest rated usability factors. 



Student Usability Ratings 




Error Rate Mental Task Time-on-Task Quality of Satisfaction Graphics 

Workload Completion Output 
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Students overall found the site to be satisfactory (8.8) and effective (8.1) with a higher need for improvement in 
terms of efficiency (6.9). 

School librarians. The school librarians that tested the site liked the graphics (8.2), were generally satisfied 

(7.7) , and liked the quality of the output (7.0). Overall efficiency (5.8) in terms of error rate (5.7), mental workload 

(5.7) , and time-on-task (6.) and effectiveness (6.1) in terms of task completion (5.2) appeared, however, to be 
problematic. 




Heuristic Evaluation. The site was examined by three researchers independently utilizing 10 usability 
heuristics. The site rated high on user control (9) and aesthetic design (8.5), moderate in consideration of visual 
clarity (6.5) and user resources (6), and low in explicitness (2.5), compatibility (3), and prioritization of functionality 
( 3 ). 




Conclusions and Recommendations 

Overall the site does not appear to be used by many of the user groups it is designed to serve. Students currently use 
the website when they are assigned to by their teachers. Teachers use it primarily for research and information about 
workshops and lesson plans. School librarians use it for professional development and resources. Overall, the data 
suggests that across users they find the site to be relatively effective (6.8 out of 10) and satisfying (7.6 out of 10) 
while overall efficiency is lower (6.1 out of 10). Students appear to like the site at higher levels than teachers and 
school librarians and we surmise this may be the case because teachers assign specific tasks within the site so they 
do not have to search for information very often. 
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Table 5 - Cumulative Usability Ratings 



EFFECTIVENESS 


Student 


Teacher 


SL 


TOTALS 


Task Completion 


7.9 


5.4 


5.2 


6.2 


Quality of Output 


8.3 


7 


7.0 


7.4 




8.1 


6.2 


6.1 


6.8 


EFFICIENCY 










Error Rate 


5.8 


5.6 


5.7 


5.7 


Time-on-Task 


7.9 


6 


6 


6.6 


Mental Workload 


7 


5.6 


5.7 


6.1 




6.9 


5.7 


5.8 


6.1 


SATISFACTION 










Satisfaction 


OO 

oo 


6.1 


7.7 


7.5 


Graphics 


00 

00 


6.3 


8.2 


7.8 




00 

oo 


6.2 


7.9 


7.6 


OVERALL 


7.9 


6.0 


6.6 


6.8 



Comparing the site on usability design standards also suggests that the overall graphic design is good but in terms of 
prioritizing information and having an information architecture where it is easy to search for and find information 
the site does not rate as high. The data allows each of the study’s research questions to be answered. 

Research Question 1: Is the website relevant and easy-to-use for users? 

The site is currently used for discrete purposes by students (for homework and assignments), teachers (research, 
workshops, and lesson plans), and school librarians (professional development) and overall out of 814 responses the 
average satisfaction rating was 8.1 out of 10. The site does appear to be relevant based on the survey and focus 
groups more for teachers and school librarians than for students (few reported having been on the site at all) but 
ease-of-use seems to be more problematic as the site is rated highly on its clean graphic design and use of color but 
reserves low marks on the ability to either understand the purpose of the site at-a-glance or find information quickly. 

RQ 2 : How does the website rate in terms of efficiency, effectiveness, and satisfaction? 

All three groups find the site usable in terms of content and quality of graphic design but do not find it easy-to-use in 
terms of efficiency as it received the lowest overall ratings on task completion, time-on-task, and mental workload. 
Furthermore, when examining the usability tasks the results were telling that none of the nine teachers were able to 
successfully find a lesson plan on the site during the usability tests and that they also had difficulty finding their way 
back to the home page (no home button on the site). Feedback from the teachers suggest that the site is not being 
used as much because they are having a hard time finding what they are looking for in an efficient and effective 
fashion. 

RQ3: How does the website compare to recommended web usability design standards? 

The overall composite score for the independent heuristic evaluations was 5.3 out of 10. Rank ordering the ratings 
suggest that the site scores high on user control (no automated videos or sounds) and aesthetic and minimalistic 
design. Visual clarity (6.5) and consideration of user resources (6.0) were rated moderate suggesting again that the 
graphic design and overall content was minimalistic and easy to download. The remaining six heuristics, however, 
were rated low including the two primary standards of compatibility (3.0) and prioritization of information (3.0). 
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Table 6 - Heuristic Ratings 

Heuristic Avg. 



4. User Control 


9 


10. Aesthetic and minimalist design 


8.5 


5. Visual Clarity 


6.5 


3. Consideration of User Resources 


6 


9. Recognition rather than recall 


5.5 


1. Consistency 


5 


8. Match between system and real world 


4 


2. Compatibility 


3 


6. Prioritization of functionality and information 


3 


7. Explicitness 


2.5 


Composite Score 


5.3 



Implications and Future Research 

The usability factors of effectiveness, efficiency, and satisfaction (ISO 9241, 1998) serve as a useful framework in 
which to conceptualize and understand the relationship between user information seeking needs and website design 
and effectiveness as it relates to user relevance and ease-of-use. This framework can be used to introduce a 
vocabulary and set of viable and measurable user outcomes that can gauge what are otherwise vague and hard to 
quantify questions such as “are our users satisfied,” “is our site being used,” or “how effective and useful is our 
site?” Designing highly usable websites is more than a professionally designed graphic exterior but rather requires 
as well a robust, relevant, and easy-to-access and use content specifically designed for its intended users. Youth 
seek more exploration than straight information seeking (Fidel et al, 1999) where adult teachers and school 
librarians require relevant content and highly efficient and effective pathways as informavores (Pirolli & Card, 1999) 
with little time for unsuccessful information hunting. The general concept of “affect as information” (Schwarz 1986 
as cited in Deng & Poole, 2010) suggests that the emotions users experience when visiting a site are created through 
a balance of aesthetic pleasure with the graphic design and the ability to either explore and/or find the information 
that is relevant and useful in as an efficient and effective manner (subjective based on the information seeker). 

The implications around this research centers on establishing a preliminary process around identifying user needs 
and then using quantitative and qualitative operational definitions of usability to test the overall efficiency, 
effectiveness, and satisfaction of users and their ability to complete typical tasks. The methodology can be repeated 
for general websites as well. 

Future research will involve further empirically testing the validity and reliability of this process and the instruments 
used so that they can serve as valid methods for measuring and testing website usability. 
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Appendix A - Usability Tasks 



Usability Tasks 



I. (Students) 

Scenario: Your teacher has asked you to go to NCWiseOwl to access various resources needed for a 
report. 

1 . Where would you go to learn how to properly cite a source? 

Critical path: home>elementary/middle/high>citation machine or citation maker 

2. Now return to the home page. 

3. From the home page, look up an encyclopedia article about Australia. 

Critical path: home>elem./middle/high>lands & people database 

4. From the home page, find the first governor of North Carolina. 

Critical path: home>elementary/middle/high>north Carolina web sites>nc governors 

5. Find information on internet safety. 

Critical path: home>kaleidoscope>nc students>web wise kids on internet safety 
Critical path: home>parents>intemet safety 
Open ended questions: 

1. Take them to the home page and ask them their overall reactions to it (like it, dislike it?) 

2. Take them to the appropriate subsection for the user and again ask them their general reactions. 

3. Take them to appropriate zones and get their general reactions to each (Kaleidoscope, Hover 
Craft, TechKnow Park) 

4. Ask them what they might like to see added to the site 

5. In general, rate the following on a satisfaction scale from 1-10 (l=lowest, 10=highest): 

a. Graphic design 

b. Use of color 

c. How about the information? 

i. Do you find it useful? Confusing? 

6. Ask them what they might change 



II. (Parents) 

Scenario: Your son/daughter’s teacher has asked you to help him/her go to NCWiseOwl to access various 
resources needed for a report. 

1 . Where would you go to learn how to properly cite a source? 

Critical path: home>elementary/middle/high>citation machine or citation maker 

2. Now return to the home page. 

3. From the home page, look up an encyclopedia article about Australia. 

Critical path: home>elem./middle/high>lands & people database 
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4. From the home page, find the first governor of North Carolina. 

Critical path: home>elementary/middle/high>north Carolina web sites>nc governors 

5. Find information on internet safety. 

Critical path: home>kaleidoscope>nc students>web wise kids on internet safety 
Critical path: home>parents>intemet safety 

6. Where would you go to find a full curriculum for 5 th , 6 th , 7 th , or 8 th grade? 

Critical path: home>kaleidoscope>browse by topic>curriculum bulletin boards> grade level 

Open ended questions: 

7. Take them to the home page and ask them their overall reactions to it (like it, dislike it?) 

8. Take them to the appropriate subsection for the user and again ask them their general reactions. 

9. Take them to appropriate zones and get their general reactions to each (Kaleidoscope, Hover 
Craft, TechKnow Park) 

10. Ask them what they might like to see added to the site 

1 1. In general, rate the following on a satisfaction scale from 1-10 (l=lowest, 10=highest): 

a. Graphic design 

b. Use of color 

c. How about the information? 

i. Do you find it useful? Confusing? 

12. Ask them what they might change 

III. (Teachers) 

Scenario: You would like to visit NCWiseOwl to access various supplemental resources to assist you. 

1 . Starting at the home page, find information about different student learner profiles. 

Critical path: home>kaleidoscope>nc educators>2. learner profiles 

2. Return to the home page 

3. Find grant writing tips and list of grant opportunities. 

Critical path: home>ebistro>grant resources 

4. From the home page, find reflective learning activities and icebreakers. 

Critical path: home>ebistro>reflections on learning 

5. From the home page, find a lesson plan: 

Critical path: home>kaleidoscope>grade 5/techknowpark>for teachers only! hut>for teachers only 
magazine>content colesium>lesson plans 

Open ended questions: 

6. Take them to the home page and ask them their overall reactions to it (like it, dislike it?) 

7. Take them to the appropriate subsection for the user and again ask them their general reactions. 

8. Take them to appropriate zones and get their general reactions to each (Kaleidoscope, Hover 
Craft, TechKnow Park) 

9. Ask them what they might like to see added to the site 
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10. In general, rate the following on a satisfaction scale from 1-10 (l=lowest, 10=highest): 

a. Graphic design 

b. Use of color 

c. How about the information? 

i. Do you find it useful? Confusing? 

1 1 . Ask them what they might change 



IV. (School Librarian/Tech) 

Scenario: You are visiting NCWiseOwl to access various resources needed to assist you. 

1. From the home page, find grade level competencies for computer skills. 

Critical path: home>kaleidoscope>nc tech. facilitators>comp./tech skills standard course of study 

2. Return to the home page. 

3. Locate local/professional conferences and dates. 

Critical path: home>media/tech zone>conferences 

4. From the home page, find a review of a technology-based instructional tool. 

Critical path: home>media/tech zone>evalutech 

Open ended questions: 

5. Take them to the home page and ask them their overall reactions to it (like it, dislike it?) 

6. Take them to the appropriate subsection for the user and again ask them their general reactions. 

7. Take them to appropriate zones and get their general reactions to each (Kaleidoscope, Hover 
Craft, TechKnow Park) 

8. Ask them what they might like to see added to the site 

9. In general, rate the following on a satisfaction scale from 1-10 (l=lowest, 10=highest): 

d. Graphic design 

e. Use of color 

f. How about the information? 

i. Do you find it useful? Confusing? 

10. Ask them what they might change 
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Appendix B - EES Post Evaluation Form 



Criteria 


Rating 


Comments 


Effectiveness: extent to which a goal or task is reachet 


1 


1. Task Completion 

Rate your satisfaction level with how effective you 
were in completing the tasks you were asked to 
perform. 






2. Quality of Output 

Rate the overall quality of the output you encountered 
upon completion of your tasks. 






Efficiency: amount of effort required 


3. Error Rate 

Rate your overall satisfaction level in terms how easy it 
was to complete each task without making errors or 
false starts. 






4. Time on Task 

Rate your overall satisfaction level in terms of how 
easy it was to complete tasks in a timely fashion. 






5. Mental Workload 

Rate your overall satisfaction level in terms of how 
easy it was to complete tasks without too much mental 
effort. 






Satisfaction - level of comfort user feels in being able to attain goals. 


6. Quantitative 

Rate your overall satisfaction level with the Web site 
you evaluated. 






7. Qualitative 

Please tell us some of the major reasons for your rating. 
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Appendix C - Usability Design Heuristics 



Heuristic 


Comments 


Rating 

(l=lowest, 

10=highest) 


1. Consistency 

Designing a product so that similar tasks are 
done in similar ways. 






2. Compatibility 

Designing a product so that its method of 
operation is compatible with users’ 
expectations based on their knowledge of 
other types of products and the “outside 
world.” 






3. Consideration of User Resources 

Designing a product so that its method of 
operation takes into account the demands 
placed on the users’ resources during 
interaction. 






4. User Control 

Designing a product so that the extent to 
which the user has control over the actions 
taken by the product and the state that the 
product is in is maximized. 






5. Visual Clarity 

Designing a product so that information 
displayed can be read quickly and easily 
without causing confusion. 






6. Prioritization of functionality and 
information 

Designing a product so that the most 
important functionality and information are 
easily accessible to the user. 






7. Explicitness 

Designing a product so that cues are given as 
to its functionality and method of operation. 






8. Match between system and real world 

The system should speak the users' language, 
with words, phrases and concepts familiar to 
the user, rather than system-oriented terms. 
Follow real-world conventions, making 
information appear in a natural and logical 
order 
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9. Recognition rather than recall 

Make objects, actions, and options visible. 
The user should not have to remember 
information from one part of the dialogue to 
another. Instructions for use of the system 
should be visible or easily retrievable 
whenever appropriate. 






10. Aesthetic and minimalist design 

Dialogues should not contain information 
which is irrelevant or rarely needed. Every 
extra unit of information in a dialogue 
competes with the relevant units of 
information and diminishes their relative 
visibility. 
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Abstract 

This study examines the design and preliminary implementation of a performance data system which utilizes the 
science of informatics in an educational systemic change setting. Collectively referred to as educational informatics, 
this performance data system is designed to provide school leaders with real-time data at mega, macro, and micro 
organizational levels. The theoretical framework, preliminary design, and how it relates to systems thinking and 
systemic change will be discussed. 



Introduction 

Educational informatics represents the intersection of three disciplines: teaching and learning, information science, 
and information communication technologies (ICTs) (Ford, 2004; Flaythornthwaite, 2006; Sheffield University, 
2011) and has been defined as, “the study of the application of digital technologies and techniques to the use and 
communication of information in learning and education” (Levy, Ford, Foster, Madden, Miller, Baptista Nunes, 
McPherson, & Webber, 2003, p. 299). In 2008, Ford further refined the definition to, “the development, use, and 
evaluation of digital systems that use pedagogical knowledge to engage in or facilitate resource discovery in order to 
support learning” (Ford, 2008, p. ix). It combines the application of information computing and technology and 
information management techniques (Kling & Flara, 2002) to the field of education and examines the role 
information communication technologies play in formal and informal learning (Levy, et al., 2003). As an applied 
theoretical construct in practice and action research, educational informatics parallels the use of informatics in other 
fields such as medical informatics, bioinformatics, and health informatics. 

The term “informatics” semantically is derived from the Russian word informatika and originally defined as, “the 
study of information processing” (Dictionary.com, 2009) and is considered a synonym for the word information 
science (Dictionary.com, 2009; Merriam-Webster Dictionary) and has been around since the 1960’s 
(Dictionary.com, 2009; Kling & Flara, 2002; Merriam-Webster Dictionary). The role of information processing or 
the science of how humans cognitively engage with information takes on similar meanings in diverse fields with a 
perhaps narrower focus on application and the technologies involved. Kling and Flara (2002) note that informatics, 

“. . . is usually coupled with some adjective, such as medical informatics, bio-informatics, chemical informatics, or 
educational informatics. These "X-informatics fields" are often defined as the application of information and 
communication technologies (IT) and information management (IM) techniques to "topic X" (Kling & Flara, 2002). 

Studying the techniques and impact of “digital technologies” within the context of the “use and communication of 
information” in educational settings represents an subtantively broad area of study. Focusing on the logical 
collection and analysis or analytics (Dictionary.com; Merriam-Webster Dictionary) of information can be 
considered an evaluation and management process seminal to the fields of systems thinking, human performance, 
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and performance technology. Chow (2008) and Chow & Whitlock (2010) introduced an operational definition of 
educational informatics within a systems framework, “how information technology is used to collect, organize, use, 
and disseminate information to support and help improve overall performance of the educational system” (Chow, 
2008, p. 51). This definition of educational informatics served as a major construct underpinning research on a 
Georgia model high school founded on systems principles, which requires continuous peformance data be rigorously 
collected, analyzed, and used for continual improvement (Chow, 2008; Chow & Whitlock, 2010). 

Conceptualizing the broader field of educational informatics within an analytics and systems thinking framework, 
moves the focus squarely on how ICTs are being used to collect data and how organizations are able to translate and 
use this data in a transformative fashion so that it becomes information they can use to continuously improve current 
and future decisions and performance in a real-time, dash board fashion. Chow and Lopez-Guerra (2011) have 
developed the Systemic Educational Informatics (SEI) model that combines the data driven framework of 
educational informatics with Lopez-Guerra’ s Impact Evaluation Process (IEP). The result is a model that posits 
what, how, by whom, and for what purpose data can be identified and collected utilizing ICTs. 

Figure 1 projects a potential educational informatics model for a public charter career and technical education 
oriented high school called the Central Educational Center: 



SEI and Systemic Change 




Economic Impact 



Outreach & 
Awareness 



Stakeholder 

Engagement 



Stakeholder 

Engagement 



Post High 
School 



Educational 

Accountability 



Economic 

Impact 



Student 

Participation 



Dual 

Enrollment 



Student 

Measures 



Work- based 
learning 



Outreach & 
Awareness 



Figure 1 - SEI and Systemic Change 
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Figure 2 projects Guerra-Lopez’s Impact Evaluation Process (2007; 2012) which provides the tactical steps in which 
to identify and build a customized organizational SEI model. 




Figure 2 - IEP Model 

The SEI model operates at three organizational levels: Mega, Macro, and Micro. The Mega level represents the 
societal level and requires that an organization measures its intended value-added impact of society itself (Of what 
value are we to society?) to ensure proper alignment. The Macro level involves the medium term goals of the 
organization and stakeholders, the more immediate or direct impact that benefits the organization itself (What does 
success mean and look like for our organization?). The Micro level involves internal building-block results of the 
organization (What short-term products have to be accomplished by teams and individual organizational members?) 
(Kaufman, 2006; 2011). 

For the Central Educational Center, the mega level is the international, national, and local community which it 
serves on a day-to-day and long term basis (what indicators would help us manage this, and there must be measured, 
what data should be collected, how frequently, by whom and for what types of decisions? What information would 
CEC need to have to make informed decisions about its current and future performance? And what goals and 
objectives do these relate to?). CEC’s macro level is more narrowly defined as the set of decisions and goals 
associated with the CEC’s survival and well-being, including charter goals and objectives. Similar to the mega 
level, a data system must be established to collect the relevant data in as automated fashion as possible. Lastly, 
CEC’s micro level involves its own internal objectives, and operational goals, as well as the unique goals of its 
internal customers (faculty, staff, and students). 

The Impact Evaluation Process (IEP) is a systematic process that articulates data within the context of value-added 
to both client and society. As part of the SEI model each goal in mega, macro, and micro will follow the same 
process: 



• Identify key stakeholders (and users of the system) for each level, and their expectations 
and purpose for the information system 

• Identify key decisions for improvement and management. Discussion about Mega, 
Macro, & Micro results critical at this point. May have to modify plans and clarify 
desired results when alignment or misalignment becomes evident. 

• Identify relevant measureable indicators. . .what will be measured, how often and why. 
Balance between effort to track the set of indicators and potential value of using them 
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must be established. Also, must set targets for indicators as well as baseline. It will be 
appropriate to establish targets first for some indicators, while others require the 
collection of baseline data before sensible targets can be set. 

• Set up systemized data collection methodology, that includes sources, instruments, 
procedures, maintenance, and responsibilities for specific team members. 

• System must be programmed for automated analysis, and simple utility. Utility must be 
embedded as part of a broader performance system. It should be aligned to users’ 
broader performance responsibly and consequences, as opposed to something separate an 
additional to what they currently do and how they do it. 

• System can be developed to provide various levels of guidance from more general 
guidelines, to more specific decision-making job aids using “If this, then that” logic. 

That is, the system could be set up to guide the user not only through the identification of 
performance gaps, but also to causal analysis, and potential solution alternatives (Guerra- 
Lopez, 2008; 2010). 

The SEI model with mega, macro, and micro indicators (See Figure 3) moves the theoretical, research-oriented 
conceptualization of educational informatics into the action and applied research setting integrating the much more 
applied concepts of informatics (the science of information) and analytics (logical organization and analysis of 
information) within systems thinking (ends, goals, and processed) and human performance technology (conditions 
of optimal human performance). CEC’s utilization of the SEI model will reflect its first real-world application, 
which is consistent and resonates with the school’s history and tradition of leadership, innovation, system’s thinking, 
and data-driven decision making for the greater benefit of students, stakeholders, and society itself. 
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Figure 3 - SEI Model 

Design and Development of the CEC Systemic Change Informatics System 

The SEI model is being designed in collaboration with the Central Educational Center to ensure validity of mega, 
macro, and micro measures. Four separate meetings were held to identify appropriate indicators, metrics, and data 
including a formal presentation in front of the CEC Board of Directors. The SEI system is being designed as a 
dashboard of critical indicators so CEC is able to view its mega, macro, and micro level performance in real-time. 
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CEC’s Mega Level Goals. 

CEC has identified three mega goals - Economic impact, outreach and awareness, and stakeholder engagement. 
Economic impact has initially been operationalized as return on investment ratio calculated through dividing initial 
capital costs by measured student benefits identified as the total number of students who are enrolled in work-based 
learning and pursuing a technical college certificate, which on average earn a starting salary of $32,000 minus the 
same number of students earning minimum wage. As an initial operational definition, this calculation most likely 
will be refined over time as the model evolves. A second indicator is dropout cost savings, which is the amount of 
money the local community saves or recoups from the state department of education that allocates a set amount of 
funding for every student who attends public schools. The preliminary measure that has been identified involves 
multiplying the number of students retained by state allocation minus the state high school dropout average. 

Outreach and awareness reflects the community’s sense of value and the importance of enrollment, marketing, and 
recruitment for charter schools such as CEC, which must recruit student to attend. The preliminary indicator that 
will be used will be community awareness and perceived value, which will most likely be measured through an 
online survey disseminated to all school councils, which are comprised of each school’s administrators, teachers, 
parents, and students. The survey will be disseminated in the early part of the spring semester so that CEC will have 
an opportunity to decide how best to utilize its marketing and publicity resources effectively. 

Stakeholder engagement represents an indicator of the overall satisfaction of CEC’s primary stakeholders - 
employers from business and industry, educator administrators, and the CEC board of directors comprised of general 
school stakeholders. Another online survey will be disseminated to the three formal boards representing the county’s 
economic development commission, chamber of commerce, and board of directors. 

CEC Macro Level Goals. 

CEC’s macro goals are defined specifically by its charter with the state board of education, which identifies a series 
of goals it must address in order to maintain its legal charter status. 13 macro goals, all charter objectives, have been 
identified as part of the SEI model which focuses on student or school performance and growth: 

1 . Student performance will meet or exceed the federal average test scores for reading and language arts. 

2. Student performance will meet or exceed the federal average test scores for math. 

3. Student graduation rate will meet or exceed the state average. 

4. An increase in students who attempt and achieve Certified Work Ready (WorkKeys) status. 

5. The Chamber of commerce has a growth goal for CEC. 

6. The Board of Education has a growth goal for CEC. 

7. The Technical College has a growth goal for CEC. 

8. Increase in physical growth of CEC or project co-location. 

9. Increase of CEC network of state-wide replications. 

10. 90% placement rate of graduates who earned a technical college certificate. 

1 1 . Students will meet or exceed state average for end-of-course test scores. 

12. 90% placement rate of graduates from work-based learning program. 

13. Decrease county dropout rate. 

All 13 organizational goals are already being measured. The current issue, however, is that the data for each goal is 
not archived or organized in such a way that CEC leadership can quickly analyze and identify potential problem 
areas that will inform future decisions. The CEO feels the SEI system will promote the use of relevant data and save 
his organization time in collection and organization of data so they can spend more time in translating it to 
information that can be used in making informed decisions. 
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CEC Micro Level Goals. 



CEC’s current micro goals are defined specifically by its legal charter to increase overall student enrollment for: 

1 . Dual enrolled students 

2. Dual enrolled students who successfully earn a certificate 

3. Work-based learning students 

4. Internships, apprenticeships, clinical, or job shadowing 

5. High school senior participation 

6. Overall enrollment 

7. Overall technical college enrollment 

Similar to the macro goals, all current micro goals are already currently measured. By bringing together this data in 
one central location, CEC will be able spend less time gathering it and more time using it to inform future decisions. 

SEI Model v.l: A performance dashboard 

Google Analytics software is able to collect and report user web activity to a granular level in a dashboard format 
that allows organizations to identify user information such as location of their web hits, the type of browser, type of 
monitor resolution, whether they were referred by a search engine or through a direct URL, how long they view 
pages, which pages they clicked on, etc. Designers are able to utilize this centralized information to make informed 
decisions about their website. 

The SEI system in close consultation with CEC is designed in a similar format. The current articulation of the SEI 
model for CEC organizes mega, macro, and micro indicators in a performance system dashboard organized in six 
columns - indicator, metric, data type, data, person responsible, and status of data collection. 



MEGA 


Indicator 


Data Source 


Data 


Responsibility 


Status 


1 . Economic Impact (return on 
investment of school, data includes 
economic development and cost 
savings for decreased dropout rate) 


1A. CEC ROI (cost vs. benefit); Initial costs for CEC 
(capital); The number of students in WBL and number 
of students in dual-enrollment. Savings would be #of 
students who save dollars by getting to the 
workplace and college earlier. Cost of the gap (avg. 
salary vs. minimum wage salary)? 


WBL Students 








Dual Enrolled Students 








CEC Capital Cost 








ROI (Cost divided by Benefit) 








1 B. Dropout cost savings - Cost per dropout x 


Pre CEC county dropout raw numbers 








Current CEC county dropout raw numbers 








formula. Dropout rate pre CEC compared to 
annualized drop out rate since CEC opened. 


State funding per student 








Dropout Cost Savings (Pre/Post comparison) 








2. Outreach and Awareness (% 
randomly selected stakeholders aware 
of the school, data will be results of an 
online survey) 


2A. CEC Board Ratings (twice a year - start of a 
board year and end of a board year; all stakeholders 
are on this); Economic Development (Development 
authority board; chamber board) - once a year; 
School leaders (Dartner schools olus all schools) 


School Council Survey (High School) 








School Council Survey (Middle School) 








School Council Survey (Elementary School) 








School Council Survey TOTAL 








3. Stakeholder Engagement & Value 
(engagement scale, data includes 
engagement scale and school leader 
ratings) 


3A. CEC Board Ratings (twice a year - start of a 
board year and end of a board year; all stakeholders 
are on this); Economic Development (Development 
authority board; chamber board) - once a year; 
School leaders (partner schools plus all schools) 


CEC Board Rating (Fall) 








CEC Board Ratinq (Sprino) 








Economic Development 








School Leaders (Hiqh School) 








School Leaders (Middle School) 








School Leaders (Elementary School) 









Figure 4 - SEI Mega Level 
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MACRO 



Indicator 



Data Source 



Data 



Responsibility Status 



1 . Meet or exceed Perkins (Federal) 
Readingfl_anguage Arts (Charter 
Objective 1) 


Already collected by state; distributed to the 
superintendent; sent via email. 


Reading CEC 








Reading Federal 








CEC Reading Differential (+/-) 








Language Arts CEC 








Language Arts Federal 








CEC Language Arts Differential (+/-) 








2. Meet or exceed Perkins (Federal) 
Math (Charter Objective 1) 


Already collected by state; distributed to the 
superintendent; sent via email. 


Math CEC 








Math Federal 








CEC Math Differential (+/-) 








3. Meet or exceed Perkins (Federal) 
Graduation Rate (Charter Objective 2) 


Already collected by state; distributed to the 
superintendent; sent via email. 


Graduation Rate CEC 








Graduation Rate Federal 








CEC Graduation Rate Differential (+/-) 








4 Increase # Certified Work Ready 
(WorkKeys) (Charter Objective 5) 


Collected internally over two semesters - increase in 
students who attempted and achieved. 


WorkKeys CEC students (fall) 








WorkKeys CEC student certified (fall) 








WorkKeys CEC students (spring) 








WorkKeys CEC student certified (spring) 









Figure 5 - SEI Macro Level 



MICRO 


Indicator 




Data 


Responsibility 


Status 1 


1 . Increase # Dual-Enrolled WGTC 


Actual and projected. Growth goals in the spring 


Actual # Dual-Enrolled 








(Charter Objective 3) 


Previous # Dual-Enrolled 












Chanqe (% +/-) 








2. Increase % Dual-Enrolled WGTC 
who complete TCC (Charter Objective 
3) 




Actual % Dual-Enrolled 










Previous % Dual-Enrolled 










Change (% +/-) 








3. Increase# Work-Based Learning 
(Charter Objective 4) 


Actual and projected. Growth goals in the spring 


Actual # WBL 








Previous # WBL 










Change (% +/-) 












Actual# 








4. Increase # intern + apprenticeship + 
clinical + shadow (Charter Objective 4) 


Actual and projected. Growth goals in the spring 


Previous # 










Change (% +/-) 












Actual# 








5. Increase H.S. Senior Participation 
Rate (during h.s.) (Charter Objective 
13) 




Actual % of county 








Annual from previous year. Projected in the spring. 


Previous # 








Previous % of county 










Change (# +/-) 












Change (% +/-) 








6. Increase High School Enrollment 


Annual from previous year. Projected in the spring. 


Actual# 








(Charter Objective 13) 


Previous # 












Change (% ■*•/-) 








7. Increase Technical College 
Enrollment 


Annual from previous year. Projected in the spring. 


Actual# 








Previous # 










Chanqe (% +■/-) 









Figure 6 - SEI Micro Level 

The SEI model is characterized by four primary characteristics: system indicators, real-time data, automation, and 
performance improvement focus (primarily gap identification and causal analysis), . The initial SEI model as 
conceptualized will provide a dashboard of real-time data around Organizational Elements Model (OEM) (Kaufman 
2006; 2011) mega, macro, and micro system layers designed in collaboration with CEC leadership. The initial 
automation will utilize a Google document which will allow various data to be entered directly into the dashboard 
while other data points will need to be manually entered. The overall value the SEI model in terms of supporting 
organizational performance in terms of data driven decision making is still undetermined and will be followed over 
the 2011-2012 academic year. 

Systemic Change and the Systemic Educational Informatics (SEI) Model 

In 1997, Joe Harless envisioned what a public school implementing his Accomplishment-Based Curriculum 
Development (ABCD) system might look like (Harless, 1998); that school, the Central Educational Center (CEC), 
opened its doors just three years later, in 2000. While not representing total fidelity to his vision, CEC represents an 
approximation of Harless’s need and customer-driven vision of education largely made possible through the 
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flexibility allowed through charter schools; the ultimate customer of public education is society itself and these 
needs inform what and how public education teaches. The major stakeholders of society could be more precisely 
identified and served: employers, parents, students, educators, and post-secondary educators. 

CEC’s charter serves as a legal contract with the state Department of Education that precisely identifies the 
accomplishments it has pledged to fulfill. Educational informatics utilizes the science of information and computing 
technology to collect, organize, and disseminate real-time data to inform decision making in the educational setting 
to improve teaching and learning on an organizational, management level. CEC leadership views this emerging 
science as a way to make identifying and gathering the data necessary to make informed decisions more efficient 
and streamlined so more time can be dedicated to analyzing and utilizing this data so it can impact real-time 
performance decisions. Reigeluth (1994) views systemic change as a paradigm shift that requires formal education 
to refine itself in order to meet the growing changes and needs of society at all levels of education. The SEI model 
represents the first applied application of educational informatics, integrating contemporary technological advances 
in computing and information science, within a systemic change context. 
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ABSTRACT: This study describes a partnership experience between a team of external evaluators and the 
stakeholders of an online graduate program at a mid- western university in the U.S. Using a Participant-Oriented 
evaluation, this study looks at how program evaluation activities lead to the improvement of an online graduate 
program. Findings suggest that the program evaluators play a critical role in (1) supporting faculty members in 
overcoming obstacles related to teaching online, and (2) facilitating dialogue among stakeholders. 

Key words: Program Evaluation, Distance Learning, and Participant-Oriented Evaluation Approach. 

PURPOSE AND IMPORTANCE OF THE STUDY 

What is program evaluation? Why do people need it? According to Wholey, Hatry, and Newcomer (2007), the field 
of program evaluation provides processes and tools that workforce educators and developers can apply to obtain 
valid, reliable, and credible data to address a variety of questions about the performance of programs. Program 
evaluation is often defined as “a systematic process used to determine the merit or worth of a specific program, 
curriculum, or strategy in a specific context.” (Gusky, 2000, p. 1 3). In other words, program evaluation helps us see 
the effectiveness and value of a program by measuring the outcomes of the program. Nevertheless, depending on the 
focus, program evaluation may utilize different approaches. In this study, we look at how evaluation activities, from 
a perspective of a Participant-Oriented approach, helped improve an online Master of Science Program in the field 
of agricultural sciences. Participant-Oriented evaluation approaches use stakeholders (instructors, program personnel 
and students) to provide assistance with carrying out the evaluation. Stakeholders are involved to achieve a variety 
of goals, such as: “stakeholders’ greater understanding and ownership of the evaluation leading to greater use of 
results or to empowering stakeholders, building evaluation capacity in the organization, and increasing 
organizational learning and date-based decision making.” (Fitzpatrick et al., 2011, p.189) 

Program evaluation can be divided into a face-to-face environment and a distance education environment. In this 
study, a distance education environment will be the main focus. Why focus distance education? Distance education 
provides access to education for people who would otherwise not participate in face-to-face traditional programs, 
allowing the learner and instructor to be physically apart during the learning process, while maintaining 
communication through various channels. Although this need increases over time, organizations which offer 
distance-delivered courses have a compelling need to determine their quality (Lockee, Moore, and Burton, 2002). 

By having ways to determine the courses’ quality, the program ensures the course quality, improves the overall 
program, and is able to attract more prospective students into the distance education program. 



362 



The program examined in this study was an online Master of Science program designed to professionals in the field 
of seed technology and agri-business who wanted to get an advanced degree in this area of specialization. This 
program, offered at a mid-westem university, is a 36-37 credits program with courses on science, technology and 
business focused on decision-making in the seed sector. This program aims at increasing its value through a 
systematic program evaluation. However, it is not always possible to evaluate online courses exclusively based on 
students’ performance (Lockee et ah, 2002). Therefore, a thorough program evaluation with a solid approach that 
leads to a greater understanding of an online education program, its strengths and challenges is needed. As a result, 
in this study a Participant-Oriented Approach was used to conduct the program evaluation with the involvement of 
the stakeholders, instructors and students. 

The program evaluation was conducted by a group of two external evaluators who involved online instructors, 
students, and program administrators in the evaluative effort. The use of the evaluative results was facilitated by the 
communication, collaboration, and strong partnership between the program evaluators and these stakeholders. For 
example, there was a great amount of focus on providing the program instructors with recommendations on online 
pedagogy and assessment techniques (Correia & Yusop, 2010) as well as strategies on better use of different 
technologies in their online courses. 



METHODS AND DATA SOURCES 

This study utilizes a case study approach (Yin, 2008) to gain an understanding of how students and instructors in the 
program explore online learning and teaching. Eleven stages were followed as a strategy to approach this program 
evaluation from a Participant-Oriented perspective: 

(1) Talked with stakeholders: The program evaluators started the whole process by scheduling initial meetings with 
the program director to talk about the plan for evaluation. 

(2) Identified program scope: In order to better help with the program, the evaluators also consulted with the 
program director for specific expectations. 

(3) Defined program activities: The evaluators participated in the program meetings and investigated course 
websites to understand the program activities. The activities in the seed technology program included two courses 
per semester and two program meetings with instructors and program personnel. 

(4) Discovered purpose of the evaluation: Through discussion with program director and instructors, the program 
evaluators understood that the program, as well as its students and instructors need to be helped to perform better in 
the online environment. 

(5) Conceptualized issues: This was combined with both stage 3 and 4; 

(6) Identified data needs: In order to provide enough feedback to the stakeholders, the evaluators made sure what 
data needs to be included in the reports. 

(7) Selected instruments: The evaluators chose the appropriate instruments to conduct the evaluation. 

(8) Observed designated outcomes (9) Thematized (10) Confirmed and (11) Assembled reports: In common cases, 
stage 8, 9, 10 and 1 1 could be combined together to produce the final report. At this final stage, the evaluators 
carefully observed the outcomes and tried to analyze the results by categorizing the results into appropriate themes. 
In this way, the stakeholders could fully understand the results and suggestions provided by the evaluators. Finally, 
during the faculty meeting, the evaluators presented the findings and submitted reports to the stakeholders. 

For data collection method, each case is based on data collected from open-ended questions in a questionnaire 
designed by Respondus from WebCT (a learning management system), interviews with course instructors and 
analysis of course-related documentation, including course syllabi, schedule, quizzes, handouts, and instructional 
materials. Data from a questionnaire with 39 questions were retrieved at the end of each semester and included items 
related to students’ online teaching and learning experiences, problems faced during the learning/teaching process, 
and specific procedures or steps for course improvement. Data were analyzed using constant comparative methods 
(Strauss & Corbin, 1998). Interviews and open-ended questions were organized into themes, which were categorized 
and coded. 



MAJOR EVALUATIVE FINDINGS 

The final evaluative findings can be categorized into instructor and student’s perspectives. Each perspective has 
different themes described in the following. 
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The Instructor ’s Perspective 



A. Communication among instructors 

In courses with more than one instructor co-teaching a need to improve communication in order to map course 
content on a regular basis was found. Such a connection is important to help strengthen the course communication 
and reduce any confusion among the students. Thus, it was also recommended that the program coordinator should 
arrange once a semester for a formal gathering such as an “Experience Exchange Day,” This activity would allow 
for past, current and future instructors in the program to share their experiences and best practices on teaching 
online. At the same time, the program coordinator could provide course templates or and other tips on online 
teaching and learning and support the program instructors on a more proactive way. He could bring experienced 
online instructors around campus to a program faculty meeting to share some strategies and tactics to teach online. 

B. Coordination among instructors and courses taught 

A need for instructors to learn more about the program curriculum and to get to know each other better in order to 
develop a sense of community was identified. The following strategies were recommended to increase coordination 
among instructors and courses taught: (1) inform instructors of the courses students have completed and need to 
complete; (2) share syllabus among instructors to avoid content overlapping; (3) gain understating of students’ 
knowledge and skills acquired in the program. 

The Student ’s Perspective 

A. Course Content: Fact-oriented, industry-linked 

According to the results, students appreciated course content that is fact-oriented and linked to industry practices. 
Therefore, it was recommended that the instructors keep and enhance this aspect of the courses by including more 
examples and industry case studies when teaching key concepts listed on the syllabus. 

B. Delivery and Instructional Technology: Lecture slides 

To respond to some students’ suggestions, we suggested instructors to either add more obvious transitions between 
the lecture slides to help students follow the content and/or use animated graphs or charts to support the lecture’s 
key points. We also recommended that the instructors shorten the lectures/presentations or break them up into 
shorter segments. 

C. Interactivity: Student-student and Student-instructor 

Interaction in the courses was mostly asynchronous (i.e. one-way) and in the form of instructor - individual student 
interaction, mostly for students to get feedback from instructors on students’ assignments or students to acquire 
clarification about assignments and homework. Therefore, the interaction was somewhat limited. Strategies to 
improve instructor- student synchronous interaction by using web conference tools, such as Skype, Adobe Connect, 
a chatting tool, or a discussion board tool provided in WebCT were recommended. By doing these, interactivity 
would increase allowing for more student-student interaction and student-instructor interaction in real time. Group 
discussions could be set up to promote in-depth conversations within the course. Moreover, using Skype or Adobe 
Connect, students and instructors could interact synchronously and remotely. Students could communicate not only 
with the instructors but also with peers from around the world using video and audio. 



CONCLUSIONS 

The evaluative findings showed that program evaluators play a critical role in (1) supporting faculty members in 
overcoming obstacles related to teaching online, and (2) facilitating the dialogue among stakeholders. The online 
Master of Science Program has grown and improved as a result of the evaluators’ efforts to have a complete 
knowledge of the program, its context, and the stakeholders’ needs in order to continuously offer useful information 
(Fitzpatrick, et al., 2011). After four consecutive semesters, the program evaluators became an intrinsic part of the 
program community. 



364 




By using a Participant-Oriented approach for evaluation, the team of external evaluators was able to establish a 
strong partnership with the stakeholders of the online Master of Science program. As a result of fruitful interactions 
during the evaluation process and constant communication, the main stakeholders (especially online instructors and 
students) were able to express their concerns and be explicit about the program’s strengths and weaknesses. In 
addition, the program administrators who worked in collaboration with the program evaluators were able to 
implement many of the evaluators’ recommendations. 
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Abstract 

An e-community of participants is being established to collaborate on a design to study and document the 
sustainability of innovative practices in educational technology. Such a community can recognize the unique areas 
of interest that each member brings to the effort. This paper reviews models that have been used successfully in the 
past and proposes a model that is collaboratively constructed and tested in the coming year by members with intent 
to report on lessons learned. The rationale, design, and theoretical framework of these multiple studies or design 
projects would be the common elements to connect the community of researchers within this project of determining 
how to document and evaluate the sustainability of innovative practices in educational technology. While each 
researcher’s context remains unique and identifying traits remain intact, this collaborative research can facilitate the 
development of a dynamic and practice-defined model to document and assess the benefits and challenges of 
technology integration or innovation adoption, which remains an ongoing need in the research community. This 
paper provides a framework for research and suggestions for establishing a springboard to develop and maintain this 
research community. 



Proposing Collaborative Research on Innovation 

"Numerous studies have documented that it is difficult to scale up promising innovations from the fertile, 
greenhouse environments in which they were conceived to the often barren contexts that exist in public schools..." 
(Dede, Honan & Peters, 2005) 

"Adapting a locally successful innovation to a wide variety of settings-while maintaining its effectiveness, 
affordability, and sustainability — is very challenging." (Clarke & Dede, 2009, p. 353) 

Educational leadership research describes the need for decentralized decision-making and institutional 
flexibility to encourage innovative practices (Ellison, 2009, Goodwin, Lefkowits, Woempner, & Hubbell, 2011, 
Knight, 2011, Lezotte & Snyder, 2011). Teachers and administrators can benefit from supports to implement such 
changes, supports that document the manner in which innovative practices are implemented and also evaluate the 
outcome of these efforts. Ellis and Goodyear (2010) describe the concept of ecological management, an ongoing 
oversight by all members of an educational community “to develop and articulate self-awareness” (p. 1 1 1) of 
individual action and group procedures to achieve the goals and mission of the educational institution. They argue 
that sustainability of new initiatives can be achieved through ecological management, an understanding of 
“[institutional] planning [and structures] in a context of uncertainty brought about by rapid and challenging changes 
in ways of encountering and working with knowledge” (p. 111). 

Sustaining new initiatives in educational settings is not a new goal. Rogers (1995) formulated a theory 
called "the diffusion of innovations" to explain the variances in the adoption of new strategies and technologies in 
various kinds of communities. While his early research focused on farming and third world communities, his theory 
has been applied to other areas such as educational and business communities. When desktop computers started 
penetrating businesses, researchers investigated initial resistance. Davis (1989) explained the Technology 
Acceptance Model (TAM) to identify specific factors such as perceived ease of use and perceived usefulness 



366 




impacting the use of innovations within the business world. Additional analysis of the phenomena of computer 
application use and adoption led to TAM2, an extension of TAM developed by Venkatesh and Davis (2000). 
Eventually, the evaluation of training or professional development to understand the benefits of a specific innovation 
led to the formulation of the Concems-Based Adoption Model (CBAM) by Hall, Loucks, Rutherford, and Newlove 
(1975), as they associated specific behaviors as evidence of people’s interest and willingness to use specific 
innovative devices and practices in classrooms. 

Innovative practices live and die in classrooms. Dede (201 1) suggests that transformation of educational 
practices and settings can occur when educators engage in the developmental process of scaling up and investigating 
(documenting and evaluating) innovative strategies and interventions under varied challenging conditions and 
factors. He describes a developmental process of scaling up and a transforming model of technology integration. 

He also suggests revising how we look at the integration of technology to better meet the needs of users. As he 
states, "... large-scale educational improvement requires more than innovations that work only in unusual 
circumstances and advances in theory; we must design and validate interventions that work at scale under a variety 
of adverse circumstances. Only this type of innovation will result in widespread usage of technology to empower 
learning and teaching." (p. 11) 

Bell, Schrum, Thompson and Bull (2008) emphasized the need for rigorous research design and studies that 
investigate the use of technology, considering the interaction of technology with content and pedagogy, within a 
defined understanding of student learning. This collaborative research study seeks to create a supportive network of 
parallel studies, unified through a rigorous research design, to investigate how specific, technology-based innovative 
strategies and innovations impact teachers and students. Such knowledge building (Scardamalia & Bereiter, 2006) 
efforts have been recognized in the literature as an effective means of solving problems and benefitting society. 

Much like prior research established ongoing communities of practice (Wenger, 1999) to document transformative 
changes in teachers’ methods and perspectives on technology’s impact on teaching and learning (Cranton, 2006; 
King, 2002; Schifter, 2008), this proposed supportive network of parallel studies intends to enact a developmental 
process to investigate, document and analyze diverse technology-based innovative practices and strategies. This 
conference session aims to formulate and create a research team that provides the support for documenting and 
evaluating innovative practices as they are implemented in classrooms. Our research goal is to ascertain 
characteristics and contextual factors that supported the implementation of a range of innovative practices, in varied 
educational settings (geographic, age/grade level, content area), identify constraints and determine future research 
directions and modifications to the research design, framework and model used for analyses. 

During a conference session, participants in this community of practice will brainstorm ideas for creating a 
model and research design to use in studying their particular implementation of a technology-based instructional 
innovation. Each participant will have a voice in this research model and study design. By sharing our voices, we 
can initiate this line of collaborative research and pilot or test this model to extend the recent work described by 
Dede, Honan and Peters (2005) and apply the developmental process to study and nurture the sustainability of 
educational technology innovations (Dede, 2011). If we experience success or overcome some preliminary 
challenges, we may seek to invite others to join this community of researchers. 

The only commonality across these parallel research projects will be the collaborative study design that is 
arrived at through researcher consensus. Each researcher’s context and focus of the study will remain unique and its 
identifying traits remain intact. The rationale, design, and theoretical framework of these multiple studies or design 
projects (Bruce, 2007) will be the common elements to connect the community of researchers within this project of 
determining how to document and evaluate the sustainability of innovative practices in educational technology. This 
paper provides a foundation, a springboard to develop and maintain this research community. 



Review of Theories and Models 

A few theories for the development of sustainable practices integrating technology in classrooms exist and 
appear to need further development or testing by practitioner-researcher teams. Researchers investigating the 
sustainability of innovative interventions and strategies note that variances in their contexts and other environmental 
factors have often undermined success once they are transferred to other settings (Clark & Dede, 2009). Jacobson 
and Reimann (2010) suggest that iterative evolution of the design and the continuous revision of relevant decisions 
(such as how to use resources, technologies and pedagogies to meet students' instructional needs) are critical to 
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sustained innovations. Such innovations yearn to be documented, investigated and evaluated in a rigorous manner 
so their benefits for students and teachers can be fine-tuned, extended over time, and expanded to other locations. 

This section reviews research models that have been or could be used to describe the sustainability of 
innovative practices, with the goal of defining a more unifying research design and model. So many of the 
underlying factors in these models have become variables for analysis that the design and underlying theories should 
serve as the source for rigor in this proposed sequence of parallel studies. 

Clark (1968) proposed four models for institutionalizing innovations (defined here as new knowledge that 
produces structural change) in higher education: 

• The organic growth model reflects the stages and processes of higher levels of institutionalization, 
professional functions & activities, and defines the status of innovation & intellectual boundaries, 
emphasizing the need to create institutions to learn innovation 

• The differentiation model reflects the need to specialize in tasks due to increasing knowledge and 
demands for accomplishing much in large institutions 

• The diffusion model reflects five consecutive stages: knowledge, information collection, evaluation, 
trial, adoption. In this model, some individuals become agents who distribute the innovative practices 
to others within and outside of the institution; unstratified 

• The combined-process model reflects the idea that innovation develops inside and outside of the 
institution, organically yielding disciplines. It acknowledges the hierarchical structure of institutions, 
and illustrates that organic growth and hierarchical structure feed one another to yield institutional 
change that institutionalizes the innovation 

In more recent literature, other models have been proposed, based on theories and on analyses of specific 
studies of technology integration efforts in classrooms around the world (Kopcha, 2010; Owston, 2007; Zhao, Li, & 
Frank, 2006). These proposed models can offer us a common perspective and method for engaging in parallel 
analyses of technology-based interventions or technology-supported pedagogical strategies whose implementation is 
long-term, and merit consideration as sustainable practices. Clark's four models appear to serve as the foundation for 
specific aspects of these more recently developed models to analyze technology integration practices within 
educational settings. Clark recognized the need to define the institutionalization of innovative practices, the reality 
that context or setting will encourage variation in implementation and design of innovative practices, and that 
sustaining innovation will evolve over time. 

Owston (2007) proposed an explanatory model based on his analyses of SITES-2 International studies of 
innovative uses of technology in K-12 settings. He provided an operational definition of sustainability of the 
innovation (implemented for more than two years without extra fiscal resources) and of transferability (innovation 
adopted in its essential form by another grade in that school or district). He also identified contributing and essential 
factors impacting the sustainability of the innovations described in these selected SITES-2 studies. Like Clark, 
Owston recognized the need to define sustainability and that institutional traits impact ongoing implementation of 
innovative practices where technology becomes a key aspect of educational settings and instructional practices. 

Zhao, Li, and Frank (2006) proposed an ecological analysis model to identify and understand what happens 
as innovative practices and uses of technology are introduced in classrooms. Like Owston, Zhao, Li and Frank 
recognized the impact of specific factors and mindsets that encourage technology integration and innovative 
practices in K-12 classrooms. Identifying these factors and mindsets, while nurturing and documenting the 
evolution of the ongoing, multi-directional interactions within classrooms, are necessary to understand and evaluate 
the impact of these innovative uses of technology and pedagogy to benefit students. Like Clark, Zhao, Li and Frank 
recognize the need to study the dynamics and interactions that encourage and support innovative practices within 
(and adjacent to) educational settings, or how the innovation is diffused (experienced, embraced, adopted, rejected) 
among all members of the educational community. It is through this complex process of analysis that researchers 
can determine the impact, viability and sustainability of technological and pedagogically innovative practices. 
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Kopcha (2010) proposed a systems-based model to describe how mentors have key roles to support 
teachers in their use of technology as they evolve their pedagogy and vision of effective instruction. He anticipates 
that researchers can pinpoint tasks, processes and cultural underpinnings which nurture professional development 
where teachers become increasingly comfortable with and gain expertise to incorporate innovative practices and 
uses of technology with minimal disruption to classroom cultures. Implementing Kopcha's model (to ascertain its 
value) appears to be the next step. Like Clark, Kopcha recognized the need to identify the gradual evolution of 
processes to address the cultural and social/behavioral aspects of how people learn and adopt new ways of learning 
and communicating. A desire to document sustainable instructional and pedagogical practices supported through 
effective technology integration continues to be explored and emphasized in more recent research (Bruce, 2007; 
Dede, 2011; Greenhow, Robelia, & Hughes, 2008). 

Greenhow, Robelia, & Hughes (2009) suggested that research on “bridging youth’s formal and informal 
learning with participatory media” (p. 248) is needed and identified potential questions or areas to be addressed. 
However, it was their focus on “changing the way academics engage in scholarship” (p. 252) that lays the 
foundation for this paper and this proposed collaborative research project. Such projects can refine and develop the 
research skills and methods that we use as a scholarly community to capture descriptions or evidence of these novel 
approaches which students use as they access technology inside and outside classrooms to communicate, investigate 
and share their understanding of content and document their new knowledge, as recommended by Scardamalia and 
Bereiter (2006). 

The theories and models we review in this paper can serve as the basis for a collaboratively constructed 
model that we can use for our professional growth, our particular specialization or focused purpose, and stages or 
processes. These models also include our current awareness of the interaction of many factors, within and outside of 
classrooms, that impact the students and teachers, and the learning or developmental growth, whether cognitive, 
interpersonal or behavioral. Moreover, these models and theories potentially facilitate the sustainability of 
innovative practices and strategies in classrooms and transformative educational settings. 

Proposed Research Plan 

The proposed plan is to gather over the next 1 1 months the collective experiences of researchers acting 
independently to implement an innovation in their unique settings. These experiences should include: 

1 . Enablers encountered during the year 

2. Constraints 

3. Which aspects of the research model worked well and which ones need modification for sustainability 

4. Characteristics and contextual factors that supported the implementation of the innovation 

5. The evaluation process of the implementation of the innovative practice(s) and/or strategies 

The evaluations should include those data collected throughout the implementation process, and should 
focus on the impact, value and merit of the innovative practices and strategies under investigation during these 
parallel research studies. These data will support the rigor of this group of parallel studies linked by a common 
research design and theoretical framework. 

Proposed Methodology 

A wiki has been established for the exclusive use of participants in the research discussion. Each participant 
will be able to access the wiki and document their own work, contribute documents as attachments, recommend 
other references, and leave comments on all pages of the wiki. It is hoped that this community of participants agree 
to collaborate on a design of how to study and document the sustainability of innovative practices in educational 
technology, while recognizing the unique areas of interest that each member of this community brings to this 
forum/discussion. 

The methodology for documenting our group's collaborative work after the conference will need to be 
negotiated around the following questions: 

1 . What will be each participant's commitment to implement innovation in their unique environment? 
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2. What should be posted, and should each participant have his or her own page? 

3. How often will participants commit to posting progress and outcomes to the wiki? 

4. Who will review the research progress, or what will be our commitment to reviewing each other's 
work? 

5. What is the projected timeline and some milestones for conducting this research? 

Proposed Evaluation Methods 

To maintain and increase rigor of any research project, the role of evaluation of the studied technology- 
supported innovative practices and strategies will inform the study design. A clear statement of research goals and 
questions lays the groundwork for designing the study and aligning the purpose of the evaluation with its study 
design. Employing a systematic approach and investment of time to explain the findings can yield evaluations of 
projects, which are viewed as well-designed and beneficial. It is equally useful to ascertain how members of the 
research team will measure (1) the process for studying the impact and investment in technology-based innovative 
practices or interventions, and (2) the indicators to determine if and how the technology-based innovative practices 
or interventions yielded benefit and value within the studied educational settings (Frechtling, 2002; McCawley, 
n.d.). 



Additionally, Davidson (2005) urged evaluation team members to (1) define relative merit, practical 
significance, and benchmarking, as well as (2) recruit resources that support effective evaluation of projects and 
programs. Of greatest interest in this proposed research study, would be the determination of: (1) the expertise- 
based values that inform the evaluative conclusions and (2) the list of truly good criteria to evaluate outcomes and 
actions during implementation of the technology-based innovative practices and strategies. Most importantly, 
Davidson suggested that the process of evaluation is viewed as feasible and accepted as a necessary aspect of 
implementing innovative practices in educational settings. 

Anticipated Results 

As researchers, we can determine which aspects of these proposed models resonate to design a study to 
measure and describe the capacity of an innovative use of technology in educational settings to be sustained over 
time. How can we define sustained use of an innovation? What criteria would we use to measure and describe its 
sustainability? How can we determine an innovation’s benefit or impact over time? Evaluating the outcome of using 
a specific innovation with students is an equally important aspect of this proposed study. Finally, the study should 
attempt to isolate (1) factors promoted the innovation’s implementation and (2) the realities that challenge the 
viability of the innovation in its educational setting. 

During a collegial discussion in 2002 of approaches and practices surrounding managing knowledge within 
educational institutions, Petrides and Nodine (2003) noted that it takes time for educators to develop the skills and 
processes that guide effective data collection, sharing and analysis that impact decisions. This discussion provided 
seven practical suggestions, as summarized below: 

1 . Build on the vocabulary and practices of the organizational culture. 

2. Focus on organizational members and their needs, pinpointing and following the energy sources within 
the organization. 

3. Make explicit the processes and patterns for exchanging information and knowledge building. 

4. Technology supports this process; it does not guide the process. 

5. Fine-tuning procedures is the path to improvement of student learning and outcomes, the desirable goal 
of such efforts. 

6. This is an iterative process that requires time, documentation, and reflection. 

7. The larger picture should influence the process and strategies taken to get there. 

Petrides and Nodine also emphasized that for effective knowledge management of educational initiatives, 
collaboration must be balanced among three elements: (1) practices for building relationships, (2) strategies for 
sharing knowledge, and (3) the processes or systematic approach needed to implement the innovation. The practices 
should actively engage a range of individuals to build relationships, which can yield communities of practice. The 
strategies need to document and exchange data, methods and ideas needed to reflect the strengths of the people in 
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the collaboration and goals of the innovation. Finally, processes need to be developed for implementing innovative 
practices using supportive technologies. The achieved balance is recalibrated as the process matures over time. 
Ongoing use of supportive technologies to connect the participants or researchers so useful information is 
accessible, exchanged and tracked is equally vital for implementation success. A major goal of this research is to 
determine such guidelines, and to establish the format of our supportive technologies so our idea and data exchange 
is accessible and facilitates the tracking and exchange of ideas as we create knowledge about specific interventions' 
implementation and sustainability within a specific context. 

This proposed collaboration of innovative implementations holds the promise of creating a strong support 
for those who seek a solid theoretical foundation and model for doing so. Through a common research design, 
rigorous evaluation and faithful reporting of enablers and constraints encountered during implementation and 
sustaining efforts, this community can enjoy the kind of knowledge building it takes to solve "wicked" problems and 
to positively impact students. By combining our strengths and sharing lessons learned with the larger community, 
we can test our collaborative model and provide a foundation for additional transformations needed to create an 
ecology of learning that incorporates sustained innovative practices that work. 
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Abstract 

The purpose of this study was to document the redesign of an existing doctoral reading course for online 
environment. This design process includes contributions from van Merrienboer’s 4C/ID model and Frick’s Totally 
Integrated Education (TIE) theory, and the redevelopment of course objectives to meet strategic goals within the 
doctoral program. 
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Design Context 



The Instructional Systems Technology (1ST) department at Indiana University (IU) had requested (and 
received) approval for a new online doctor of education (Ed.D.) degree program, and there was a defined need for 
existing classroom courses to be converted to an online format. The researchers identified a number of existing 
residential courses that would require conversion, and selected a doctoral reading course, with a second choice of an 
introductory course in research methods in 1ST. The researchers began the conversion of the referenced reading 
course, R711, in Fall 2010. The anticipated start date for the online course is Fall 2012. This design case documents 
our design process in converting the residential course to an online format, including rethinking of core course 
objectives, and methods to effectively meet course objectives in an online setting. 

The existing residential course intends to familiarize 1ST doctoral students with seminal and current 
readings, and to help students construct frameworks for their continuing development as scholars in the field. 
Students enrolled in this course have either a Master’s degree in instructional technology or have previously taken 
core courses in 1ST. This reading course covers a variety of topics in the 1ST field, including instruction and 
learning, design and development theory, systems, evaluation, and human performance technology. The residential 
section of this course has historically been offered in the fall semester, with 10 to 20 students enrolled, including 
both 1ST doctoral students and other students minoring in 1ST. 

In the residential R711 course, students meet once a week to discuss various seminal topics in 1ST with 
readings selected by the instructor and the department. Additional course tasks include the critique of selected 
readings and the completion of a literature review on a topic within the field. The existing course included an 
established syllabus, set of course goals, and associated list of required readings. The goals of the course included: 
1) develop an understanding of key issues and concepts within 1ST; 2) develop the ability to critically analyze and 
synthesize IST-related publications from a variety of perspectives; and, 3) develop a literature review for a topic of 
interest within 1ST. Due to the unique role of a reading course in establishing an overview of an entire field, a 
variety of readings, drawing from handbooks, seminal books, and journal articles were included in the list. Also, due 
to the rapid progression of the literature, the reading list had been modified multiple times in the past five to ten 
years to add more recent readings or reflect new trends in the field. 

The design team for this project included four first-year 1ST doctoral students (as of Fall 2010), one 1ST 
post-doctoral student, and the chair of the 1ST department, who served as the key stakeholder for the project. Three 
members of the design team had significant previous experience in instructional design settings, and the majority of 
members also had teaching experience. Three members were in the process of taking the residential course when the 
redevelopment work began, and the remaining doctoral student took the course during the second year of the course 
redesign effort. 

The primary setting of the design process was a weekly group meeting, where progress was discussed and 
tasks were assigned. During each of these meetings, starting in November 2010, detailed notes were taken in a 
Google Docs document, allowing a primary note taker to capture the contents of the meeting and any applicable 
design decisions, while other group members could log into the same document to add comments or additional 
notes. This capability allowed for ongoing triangulation of data captured from the primary work sessions on this 
project. The project goals for this research were originally directed towards formative research (Reigeluth& Frick, 
1999), but team members ensured that the documentation of the project was flexible enough to allow for a variety of 
methodological discussions. In conjunction with the goals of formative research, a design method was identified to 
structure the design process. The method selected was based on the book Ten Steps to Complex Learning (van 
Merrienboer&Kirschner, 2007), and the associated Four-Component Instructional Design model (4C/ID) (first 
introduced as a model in van Merrienboer, 1997). 

Designing the Course 

The general timeline of our design process spanned from Fall 2010 to Fall 2011, moving from initial 
concept and data gathering to identifying authentic tasks to rapid prototyping of potential structures to finalization of 
the core concepts and course structure. Each stage of the design process will be explained further in the following 
sections. 

Initial Concept and Data Gathering 

Due to the departmental need for an online version of the existing doctoral reading course, our end goal 
was defined early in the process. The doctoral reading course was selected among other potential courses due to its 
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placement early in the doctoral program, and because the department chair leading the design process was not 
involved in the everyday teaching of this course. 

The project began as an evaluative process, including the administration of a custom course evaluation 
instrument (Frick, Chadha, Watson, &Zlatkovska, 2010) to the Fall 2010 residential course. This evaluation was 
administered in December 2010, and individuals from the residential course were asked to participate in a short 
interview to provide information about their experiences in the course. Nine students were interviewed about their 
experiences in the course, including questions about potential improvements, the effectiveness of teaching strategies, 
and the role of the course in helping them solve real-world problems. In addition, two former instructors, each with 
at least three years of experience teaching the course, were interviewed in our group meeting. The faculty answered 
a set of questions similar to those posed to the students, with additional followup questions about the planning of the 
course and what teaching strategies had worked most effectively in the past. 

Ten Steps 

After solidifying the concept of an online doctoral course and parallel to collecting data about the current 
iteration of the residential course, formal development of the new course structure began. Early in the Fall 2010 
semester, the design team had decided to use Ten Steps to Complex Learning (van Merrienboer&Kirschner, 2007) to 
structure the design of the course. This decision was made due to several factors, including interest by several team 
members in applying this newer design methodology to a real-world problem, lack of case studies exploring the real 
world use of this model, and the presence of complex learning elements in the course that the group chose to 
redesign. 

The first step of the model, designing the learning tasks, was the most daunting. These tasks, based on the 
4C/ID model, were intended to be authentic whole tasks. In other words, the tasks should be ones that professionals 
would execute in the real world (authentic) and indicate a set of actions that would be representative of a complete 
task (whole) performed in the real world, not an isolated set of procedures. The group struggled to find a single 
authentic, whole task that met these criteria. The first approach was group brainstorming, with an output of 
recommended whole tasks that could be used in the final course. Several group members presented their concepts, 
many of which were based on goals for the course referenced in the interviews by previous faculty. Some of these 
concepts included the importance of academic reading skills (using the seminal book by Adler & van Doren, How to 
Read a Book), how to talk as an academician, and the sequence of reading research with understanding, critiquing 
the reading, then presenting the findings to others. 

In the design discussion, the team agreed that the student executing the real world task in the course was a 
researcher or researcher-practitioner applying their knowledge of the field to new problems or literature. This profile 
represented our target audience for this design, and is consistent with the goals of the doctoral program in which this 
course is placed. With this conclusion regarding the target audience, the design team then turned to the competencies 
of a researcher to understand and synthesize research literature, including the primacy of understanding the 
knowledge claim. While consistent structures exist in the text of most research literature — literature reviews, 
methods, data collection, and analysis — the core of each article includes knowledge claims the author was making, 
either based on previous research or their own research. The design team progressed in thinking about how to know 
whether the claims an individual author is making should be believed, and on what basis they should be believed. 

During this process to find the one whole task that would inform the course design, several weeks were 
spent discussing theories of knowing, including the theory of Totally Integrated Education (TIE, see Frick, 2011) 
and its basis in the work of Peirce (1932), Short (2007), Maccia (1987; 1988), and Steiner (1988). Although these 
materials were beneficial, the design team felt a conflict between TIE and our design process to that point, but a 
hybrid of the two approaches was attempted. In retrospect, this investigative look into theories of knowing was 
helpful, but drew the team away from the core goals of the course, and caused the team to lose context with the 
larger deliverables in the course. In particular, this focus indicated a shift toward the more cognitive aspect of the 
course, meanwhile neglecting the holistic vantage point we had begun exploring early in the design process 
(especially in initial brainstorming). In further exploration of the theories of knowing, the research team developed a 
flowchart of questions that would need to be asked regarding an individual knowledge claim in order to know 
whether that knowledge claim is justified. Also included in this inspection of a knowledge claim is the role of 
personal experience in validating or invalidating the claim. This flowchart explains the process a student might go 
through for each identified knowledge claim (Figure 1). During the Spring 2011 semester, the work-in-progress for 
this course redesign was presented at the IU 1ST Conference roundtable, and additional feedback was solicited from 
the attendees. Their feedback included discussion of what whole tasks might be appropriate, and the necessity of 
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understanding which articles were appropriate or not for beginning researchers (and what basis on which something 
should be judged as appropriate). 



I 

That-One 

(unique object or 
phenomenon) 



Recognitive 



Acquaintive 



Appreciative 




yes 



Do I believe it? 



DONE 



If yes, justify why you do. 

If no, justify why you do not. 





Philosophical 



Criterial 



Figure 1. Flowchart of analyzing a knowledge claim. 



Pilot Testing 

Once the flowchart was in draft form, the design team conducted a pilot test using a controversial article 
from the doctoral reading course syllabus. Each team member read the article individually, highlighting each 
knowledge claim they could find. In the next team meeting, the team members formed a consensus about which 
knowledge claims were foregrounded, and came to the conclusion that there were one or two primary claims in most 
articles, and then a large number of secondary claims that supported the primary claims. The design team then 
progressed through the flowchart using the identified primary claims, discussing the levels of application to 
educology (having to do with education), personal belief, and category of the knowledge claim. 

After this pilot testing, the team began to consider the second step in the 4C/ID model: sequence task 
classes. In the 4C/ID model, a task class is created for each real world task (or subsidiary whole task) and individual 
iterations of that whole task are sequenced from easy to hard. The team first separated the flowchart model into four 
task classes: identifying the knowledge claim, belief of the claim, category of knowledge claim, and type of 
knowledge claim (within the category). At this juncture in the design process, the semester was drawing to a close, 
and although we had identified a tenable whole task and related task classes, we had really just worked through the 
first or second defined step of the 4C/ID process we originally intended to follow. Although the focus was on the 
whole task and task class concepts, inevitably, discussion included other elements of the design process not 
explicitly referenced in the early stages of the defined design process. These included practical scheduling 
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considerations within the planned semester of coursework, the role of task classes that interacted with each other (or 
were sequenced against each other), and potential delivery methods for the final instruction. 

Organization and Prototyping 

The summer work session for this project included the chair of the department, still serving as a key 
stakeholder, and two members of the design team. Each of these members had taken the residential course in Fall 
2010 and had previous instructional design experience. The first two weeks of the summer served as a planning 
period, including the creation of goals, a project timeline, and initial work towards the organization of materials 
within task classes. Similar to the design group meetings during the semester, notes of what tasks were 
accomplished were taken in Google Docs to establish progress and accountability throughout the summer. Early in 
the design process, the team became acquainted with a text often used in social science research by Booth, Colomb, 
& Williams (1995) entitled The Craft of Research. This text generated additional discussion between the summer 
design team regarding reading research literature, and in particular, clarifying the role of knowledge claims 
independently from the team’s experience with TIE theory. This text was used as a reference for the remainder of 
the summer, and was included as a recommended reading in the overall course structure. 

The initial summer timeline included design and development work on the site, with completion of the 
main design tasks by early July 2011 (Figure 2). In conjunction with this timeline, a plan for creating task classes, 
supportive materials, and just-in-time (JIT) materials was detailed. In addition, a course site housing this information 
was planned, to allow for quick adoption of the course and related materials once the course was offered in the 
online format. 
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Figure 2. Development Timeline for Summer Work 
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In parallel with the development of the project timeline, existing reading materials for the course were 
quickly evaluated and mapped against the task classes the group had defined during the spring semester. Several 
major design decisions were made during this brief period of rapid prototyping, including: merging related reading 
themes from the residential course into single blocks of content (Design and Development, Learning and Instruction, 
etc.) and adopting the theme structure as an organizing concept. The second decision, utilizing themes as an 
organizing structure in the course, was made in order to present some continuity of thought throughout the course. A 
“spiral” structure was discussed, including reading an article from the majority of themes each week, but the lack of 
congruity between readings, along with minimal opportunities to critique opposing viewpoints offered by reading 
multiple perspectives within a single theme, outweighed the potential for cross-theme evaluation and exploration. 

The organization of the articles and themes against the task class structure took place over a two-week 
period at the beginning of the summer, following a rapid prototyping method. This method emphasizes quick 
exploration of multiple “what if’ scenarios and allows for user feedback without the need for full development, 
allowing a wide range of design possibilities to be explored without an overwhelming time commitment. Rationale 
for sorting the themes included potentially applicable content to focus on the unique goals of each task class. Since 
the explication of “how to” claims (located in the final task class, categorization of knowledge claims) was only 
present in a direct sense in the design and development literature, those themes were moved toward the last task 
class, with easier readings or readings with more direct knowledge claims placed closer to the first task class. In 
association with the reorganizing of the materials against task classes, readings were identified as being directly 
associated with the task class (following the strategy flowchart of knowledge claims, belief, and categorization) or 
as supportive or just-in-time information, based on the content of each reading (see Figure 3 for a sample task class). 
Some readings were recommended for removal due to age or the emergence of newer trends in the field that were 
important to explore, or lack of application of the reading to the strengths of the task class structure. Similarly, some 
readings had been revised in newer editions, and these more recent versions of the content may offer additional 
currency to the course. 
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Figure 3. The first task class, representing one theme of readings. 



After several iterations between the two primary designers, a proposed set of task classes was presented to 
the stakeholder, and was provisionally approved. Several features existed in the provisional task classes, including 
the presence of new content to more directly align with the needs of the task class, updated editions of content where 
available, and the categorization of some readings as optional or supportive information rather than as required 
reading that contributed directly to the faded task class. The supportive information was seen as a flexible concept, 
since students in the course come from a variety of educational backgrounds, and may have a wide range of previous 
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knowledge in various fields discussed in this course. Therefore, while some students may need more foundational 
readings on basic concepts, other students may already have a firm grasp of these concepts and may find some 
supportive readings less helpful. 

In addition to the design timeline and task classes, a proposed semester schedule for the class was also 
created, attempting to map each theme and its underlying task class onto a specific week (Figure 4). Major 
milestones for critique submissions or literature reviews were also defined, as well as special topic discussions at the 
beginning of the semester and student presentations of their literature review findings at the end of the semester. 
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Figure 4. Provisional semester course schedule. 



After identifying a structure for the contents of the course against the predefined task classes, additional 
investigation into the types of supportive and JIT information was needed. While this represented a natural next step 
according to the 4C/ID model, it also represented a practical step toward completing a section of the course in order 
to conduct a more thorough evaluation. One of the first design decisions within this task goal was to provide student 
and professor support by identifying knowledge claims in all articles used or referenced in the course. Practically, 
supportive materials could be built on top of knowledge claims throughout the course, since the first task class began 
with the goal of identifying knowledge claims, and all subsidiary task classes relied on a knowledge claim to begin 
the analysis process. The stakeholder decided that identifying these baseline knowledge claims was critical to 
understanding what type of supportive information (or sequence of fading supportive information) would be most 
helpful. The two designers split the readings based on their respective research interests and areas of expertise, with 
one designer addressing Learning and Instruction, Technology Integration, Systems, and Analysis & Evaluation. 
The other designer addressed the Design & Development, 1ST Methodologies, and Organization themes. Each 
designer worked on the project materials using a shared Dropbox folder, allowing constant communication as to 
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which articles had been completed and to provide version history of the documentation effort. Each article was 
annotated using Adobe Acrobat, with a combination of highlighting and comments to identify knowledge claims. 
During this identification of claims, it became clear that triangulation for this process was needed, as each designer 
tended to identify claims that were most interesting to them personally, or the claims that were most articulated by 
the article (as opposed to claims from cited materials). To address this issue, the stakeholder agreed that the 
designers should review each other’s claims after each reading was complete. 

Roadblock 

During the identification of claims process, in the first few weeks of the summer, one of the primary 
designers was diagnosed with mononucleosis, and was now unavailable for the remainder of the summer. Since the 
triangulation process had just begun, and issues were yet unresolved as to whether all identified claims should be 
highlighted or just the primary or dominant claims in the article, the project reached an impasse. After this 
roadblock, the process moved more slowly, and the design process increasingly revolved around the identification of 
claims, ignoring the larger context of the supportive materials these claims were intended to inform. In addition, the 
loss of design knowledge from the absent designer complicated the process, and resulted in a change of strategy to 
make the best of use time available. 

The “Box” 

Close to the middle of the summer, an incoming doctoral student with little background knowledge of 1ST 
was identified. This student volunteered to review the articles and related annotations and complete part of the 
flowchart with each article as a point of triangulation. After one of the designers became ill, the emphasis on a 
digital workflow (with planned efficiencies in creating final supportive materials) was reduced, primarily due to lack 
of communication by that designer with the rest of the group in relation to these goals. Eye fatigue and lack of 
readability of some of the materials on a digital screen took their toll, and the decision was made to transition to 
paper copies of each article to complete the annotation process. A cardboard box was obtained to house these 
articles (39 readings in all), using tabs to represent each theme or task and a hanging folder to represent each reading 
within the task (Figure 5). This box was organized by the remaining designer, and then passed off to the student 
volunteer for review. Although the paper articles were found to be helpful in reducing eye strain, the remaining 
researchers also found the box a helpful physical gauge of what articles had been completed, and what articles still 
remained to be annotated. 

The student volunteer read each article, agreeing or disagreeing with the annotated knowledge claims, then 
proceeding to the next step in the knowledge claim flowchart, stating their belief in the claim (or lack of belief) and 
offering supportive evidence in the form of vicarious or lived experience. After conferring with this student at the 
beginning of the Fall 2011 semester, it became clear that this process was difficult in many cases, especially when 
attempting to respond to each subsidiary knowledge claim identified in the article. The student used a strategy 
similar to that identified by the designers early in the Spring semester, clustering sub-claims under one or more main 
claims. The volunteer found it much easier to work through the materials, identifying this “main claim” from a 
cluster of supporting sub-claims, then analyzing that main claim through the lens of belief. 

Near the end of the summer, two faculty members met with the key stakeholder to discuss the progress of 
the course redesign. The initial concern preceding the meeting was the delay in discussing the progress of the 
redesign efforts. This delay was due to scheduling conflicts and the slower pace of the design team after the exit of 
an ill team member. The stakeholder presented the task classes as a the core of the new course, but the new structure 
of the course was not readily apparent — it was too complex to be easily understood, especially the design team’s 
path to identifying the “whole task.” In addition to this complexity, it was unclear from the design process what the 
instructor’s role would be in the online or residential setting based on the new structure. In other words, how would 
the instructor teach the class, and how would it parallel or deviate from the existing course structure? This served as 
a design failure, in that the team did not solicit and receive feedback from former and new professors teaching the 
course early enough in the design process once the structural elements of the course were identified. In addition, the 
results of the meeting, in addition to later conversations discussed in the next section, revealed the narrow focus of 
the redesign efforts at this stage, especially in regard to implementation in the classroom and online environment. 
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New Semester, New Instructor 



The Fall 2011 semester began with a new urgency for the completion of this course redesign for the online 
setting. In August 2011, the department received final approval for the online Ed.D. program, and applications 
began to be received for the Fall 2012 semester. This new course was slated to be offered during the first semester to 
new students as a foundational doctoral course. Changes within the residential course were also in progress, with a 
new professor teaching the course during the Fall 2011 semester. This new professor was experienced, and had 
taught a wide range of courses in the past, but had not taught this course previously. She had been part of the 
meetings with the key stakeholder during the summer design process, and sought out one of the design team 
members from the research group when planning the course syllabus in late August 2011. 

The residential course requirements were set to remain largely the same, with similar deliverables and 
readings as in previous semesters. However, careful attention was given to the outcomes of the design process thus 
far, particularly in the importance of recognizing knowledge claims to complete a critique of an article. The 
conversation about the course transitioned into a design discussion, establishing which findings from the design 
process thus far would be most beneficial in a residential setting, and would prove most effective with that particular 
professor. The main focus of discussion was the readings, and how to most effectively cover the materials during the 
face-to-face class. The second focus was the deliverables of the course, a topic that had been pushed aside early in 
the design process, but was quite important to address in the residential context through the lens of our design 
process to that point. 



[Article Name 
My name 

Article Reference in APA 



Knowledge 


• Knowledge claims made by the author in the 
reading 


Support 


■ Support for the claims you listed 




Figure 5. The box used to contain readings with 
annotations and student review comments. 



Figure 6. A proposed layout for the article critique 
and the final article critique template. 



A number of readings had been targeted over the summer, both in terms of locating primary claims, and in 
identifying newer or more appropriate articles for the goals of the course. During the design discussion with the new 
residential professor, the standard reading list (as used in previous versions of the course) was adopted, with any 
changes to be addressed on a week-by-week basis. Although some of the updated readings may have been helpful 
from the perspective of recency, it was decided to stabilize the course along this dimension and focus more actively 
on the application of these readings within the context of the in-class critique and discussion. One of the most 
important indicators that a student had actively read the articles in previous versions of the course was the 
completion of a required article critique on one article each week. While this critique was helpful, a more targeted 
critique requiring the student to interact with individual knowledge claims was found to be more helpful. Therefore, 
a new article critique template was developed (see Figure 6), requiring the student to identify a primary claim from 
one article each week, then support that claim using a source from the article’s references, and a source that the 
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student has identified independently. In addition to this critique document, each student was required to present an 
assigned article (using their critique as a basis of discussion) once during the semester. The concept of fading and 
task classes as identified in our summer design process was largely discarded, with the repetition of tasks seen as too 
cumbersome and time consuming. In addition, the ability to focus on single articles for a student critique allowed 
time resources to be spent in a more targeted way. 

While fading, as a concept for sequencing the readings, was discarded as planned in the design process, it 
was used in a more powerful way in sequencing the deliverables for the course. The previous residential course 
required weekly critiques, a minor literature review (approximately 5 pages) at the midpoint of the semester, and a 
major literature review (approximately 15 pages) at the end of the course. These deliverables had been disconnected 
in the past, and required independent research to complete each task. In this planning session for the Fall 2011 
semester, however, these deliverables were rethought and linked together to establish one chain of tasks with 
decreasing support and increasing complexity. The critique has already been discussed, with the goal of identifying 
and supporting knowledge claims both through existing article references and the location of an independent source. 
This idea was expanded for the minor literature review, replacing a formal literature review with an annotated 
bibliography. Each entry in the annotated bibliography could pull information directly from a critique, similar to 
what was being completed each week, with each critique introducing a new source to be analyzed. The topic for the 
major literature review would be decided by the student and professor earlier in the semester, so that the annotated 
bibliography could serve as a planning mechanism for the major literature review due at the end of the semester. 
This sequence of tasks allowed an integration of classroom critique and primary deliverables in a new way, based on 
the concept of knowledge claims. 

Once this structure had been planned, the semester began. One of the design team members observed the 
new residential course to see how the feedback from the design team was implemented in a face-to-face context. In 
addition, that same design team member was taking a reading course in another department that served similar goals 
as the 1ST doctoral reading course, which provided additional perspective on the standard areas of focus for this 
genre of course. The observation of the 1ST course brought out some primary pathways of working through the 
knowledge claim and support process, including the following strategy: find a claim, locate support (either internal 
to the text or through an article reference), and judge the claim based on the support provided. This pattern of 
finding knowledge claims and supporting them (with the required support needed seen as a function of the 
believability of that claim) was seen as the primary critique activity carried out in the classroom environment, and as 
a primary generator for in-class discussion. This same pattern was also emphasized in the individual critique 
documents, which also served as a generator for class discussion. 

The external readings course was also seen as a source of inspiration, in that it focused on the importance 
of knowledge claims and the related epistemology and ontology that the claims spring from. In this way, this course 
mirrored many of the goals identified in our design process, albeit in different terms, and served as a validation that 
our design process was directionally appropriate. 

Prepping for Online 

Early on in the design process, during the Fall 2010 semester, the design team discussed aspects of online 
education that could work in supporting the course under consideration. The discussion included common course 
elements such as discussion forums, chat, and wikis, as well as more collaborative tools like Google Docs, 
screencasts, and computer-adaptive supportive materials. 

One of the most important activities identified by the design team as vital for inclusion in the online course 
was the process of critique. While the face-to-face version of critique is easily accomplished, establishing online 
dialogue in this sense, particularly when students cross multiple time zones with a variety of technology capabilities, 
can be quite difficult. For this foundational concept of critique, various methods have been discussed, which include 
a fading of support and increasing difficulty. One possibility might include a webcast/screencast by a professor that 
includes a demonstration of the critique process using an article the student has been required to read. Then, the 
students might work in small groups or individually to write a critique then present it using a YouTube or Adobe 
Connect video presentation, mirroring the presentation component of the residential course. This demonstration 
component not only strengthens the critique competencies of the individual presenter, but also reinforces good 
critique behavior for the rest of the students in the course. In addition, tools such as YouTube annotations could be 
used to allow students in the course to interact with the video presentation in an asynchronous way, asking questions 
or locating issues that could be further discussed in a Google Doc or forum thread. 

The additional deliverables for the course, including the weekly critiques, annotated bibliography, and 
major literature review, could be easily accomplished in the online setting, using support for individual article 
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critiques (as discussed above) to inform these larger deliverables. Establishing lines of communication with 
residential students in the online setting (assuming online and residential courses were offered in the same semester) 
could also create a larger student community for sharing of knowledge claims and article annotations, as well as 
additional opportunities for discussion and support. 



Final Design 

While the final design of the course is not yet complete, this design case represents the process of 
developing the course structure over the period of approximately one year (through October 1, 2011). There are a 
number of specific design decisions that will still need to be made, including how students and faculty will interact 
in an online setting, what materials and methods will be used to facilitate this interaction, and how the final 
interaction experience might mirror or diverge from the defined residential course experience. 

This design process has served to finalize a course framework, not only identifying which authentic tasks 
are most important to meet specified course objectives, but also creating activities and environments to support these 
tasks in both residential and online contexts. While this design process took much longer than we originally 
anticipated, one of the biggest challenges was to identify and sequence whole tasks for this particular R711 readings 
course. This process should go more smoothly for courses already organized around tasks, such as the R690 class 
(Application of Research Methods to 1ST Issues). Nonetheless, the process of shifting from a topic-centered course 
organization to a task-centered organization may be one of the biggest challenges in attempting to use the 4C/ID 
model to guide the design process. 
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Digging In: Designs that Develop Intersubjectivity in Course Room Discourse 
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Abstract 



The purpose of this roundtable discussion was to explore factors that influence the design of the initial discussion 
prompts in course-based, online learning. The initial prompt is one of the first pieces of scaffolding necessary for the 
knowledge construction requisite in a constructivist learning environment. As a means of stimulating conversation 
among conference participants, intersubjectivity was asserted as a quality standard that is different from the current 
state of discourse in threaded discussions. Intersubjectivity is the representation of knowledge construction achieved 
through a synergistic progression from individual contributions to sequences of interdependent contributions. The 
highly interactive format of a roundtable was sought in order to elicit feedback on what participants had found to be 
effective or ineffective designs in facilitating deep interaction and knowledge construction with the expectation that 
the roundtable conversations about best (and not-so-best) practices could inspire additional points of consideration 
for disciplined inquiry in the future. 

DIGGING IN: DESIGNS that DEVELOP INTERSUBJECTIVITY in COURSE ROOM DISCOURSE 

The initial prompt is one of the first pieces of scaffolding necessary for the knowledge construction 
requisite in a constructivist learning environment. Intersubjectivity is the representation of that knowledge 
construction achieved through the synergistic progression from individual contributions to sequences of 
interdependent contributions. Instructional designers should continue to recognize that the strength of the 
construction of knowledge, as evidenced in the threaded discussion, depends upon the strength of the scaffolding 
that supports that construction. This brief paper connects relevant theory to the practice of designing initial 
discussion prompts, summarizes the importance of intersubjectivity in threaded discussions, offers evidence of the 
current state of course room discourse, and suggests implications for instructional designers. 

Connecting Theory and Practice 

According to Lim (2004), the communication that occurs in any learning environment is the most important 
aspect of the educational process that happens in that environment. Lim’s assertion is consistent with social 
constructivism (Vygotsky, 1978), which acknowledges that interaction through dialogue is crucial to cognitive 
development. Since the majority of the dialogue in the online learning environment occurs through the discussion 
boards (Jeong, 2003; Schwartman, 2006; Thompson, 2009), learners who engage these discussion boards should be 
able to achieve a high level of cognitive processing (Thomas, 2002). 

The term intersubjectivity has been used to describe the result of coordinating cognitive perspectives within 
the discourse of online learners (Dennen & Wieland, 2007). An effective means of determining the level of 
cognitive process is Bloom’s taxonomy (Jorgensen, 2009). Effectively planning the use of discussion boards is 
important in achieving this high level of cognitive engagement (Tu & Corry, 2003), and one element of planning for 
cognitive engagement is the design of the initial discussion prompts (Asherian, 2007; DeLoach & Greenlaw, 2007). 

Why Intersubjectivity is Important 

Given the role of the threaded discussion in the construction of knowledge that occurs within an online 
course (Calvani et ah, 2010), the interaction that occurs within threaded discussions is important to achieving the 
learning objectives of instruction situated within a constructivist environment. Successful knowledge construction 
requires “active and broad participation” (Sing & Rhine, 2006, p. 254) occurring at a higher level than surface 
interaction, as noted by Dennen and Wieland (2007). Knowledge construction at this more advanced level of 
interaction occurs through the opportunities for cognitive engagement required for the higher-order learning 
processes indicated by Bloom’s taxonomy (Bloom & Krathwohl, 1956). Interaction alone neither produces nor 
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demonstrates knowledge construction consistent with a constructivist perspective. Thus, while interaction is 
inherent in constructivist learning, it is feasible to take interaction to a higher level known as intersubjectivity 
(Dennen & Wieland, 2007; Martin, Sokol, & Elfers, 2008). 

Intersubjectivity represents the higher quality of synthesis represented in interactions needed to achieve the 
knowledge construction required in a constructivist environment and can be seen as the representation of knowledge 
construction achieved through a synergistic progression from individual contributions to sequences of 
interdependent contributions.. Intersubjectivity relates to the coordination of individual contributions during the 
activity, thereby creating “continuity in activity progression” through “building on each other's contributions” 
(Matusov, 1996, p. 41). Similarly, Bober and Dennen (2001) defined intersubjectivity as the development of shared 
understanding that relates one situation to another, relying on artifacts created by the ongoing conversation to 
develop new contributions to the discourse. 

Current State of Course Room Discourse 

However, as shown by the following studies, course room discourse among learners has consistently lacked 
this higher level of quality. Rather than the “sequences of dependencies” (Suthers, 2006, p. 4) required for 
intersubjectivity, researchers categorize student contributions as distinct presentations (Henri, 1992), information 
exchange (Salmon, 2000), exploration (Garrison, Anderson, & Archer, 2001), shared stories (Romeo, 2001), serial 
monologues (Pawan, Paulus, Yalcin, & Chang, 2003), consecutive online notes (Hewitt, 2005), or superficial 
postings (Bures, Abrami, & Schmid, 2010; Ke, 2010). The learners themselves share this disappointment in the 
quality of online discussions, according to Chang (2003), who found that two-thirds of students considered the 
discussions to be of insufficient value in supporting their learning. 

Implications for Design 

If intersubjectivity represents the higher level of quality needed to achieve the knowledge construction 
required in a constructivist environment (Hall, 2010), then the opportunity certainly exists to identify those elements 
which influence intersubjectivity within these course room discussions (Boulter, 2010; Wang, Woo, & Zhao, 2009; 
Wmck, 2010). As online course delivery continues to grow (Allen & Seaman, 2010), instructional designers will 
have increasing opportunities to design learning activities that promote quality course room discourse. Rather than 
relying on heuristics (Silber, 2007; Woo & Reeves, 2007), instructional designers would benefit from the 
development of research-based principles on which to design the prompts directing the initial course room 
discussions. 

In conclusion, intersubjectivity is the representation of the knowledge construction achieved through the 
synergistic progression from individual contributions to sequences of interdependent contributions. As one of the 
first pieces of instructional scaffolding presented in threaded discussions, the initial discussion prompt holds an 
important place in the design of online, course-based learning within a constructivist framework. Design variables 
that are correlated with or influence intersubjectivity within threaded discussions are worthy of additional research. 
The purpose of the interactive, roundtable discussion at this conference was to share and solicit information about 
these design variables. 
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This article provides the overview of AT & T-sponsored educational research project and the evaluation plan of the 
3D visual programming use in the project. The primary’ focus of this project was to develop and evaluate 3 D 
programming workshops designed to improve confidence, performance, and attitude toward school and learning of 
“at risk’’ high school students in the urban areas of Gotham County in an East Coastal area, where numbers of 
unsuccessful high school students has become serious concerns for educators. 

Introduction 



This article provides the overview of AT & T-sponsored educational research project and the evaluation 
plan of the 3D visual programming use in the project. The primary focus of this project was to develop and evaluate 
3 D programming called Alice, workshops designed to improve confidence, performance, and attitude toward school 
and learning of “at risk” high school students in the urban areas of Gotham County in an East Coastal area, where 
numbers of unsuccessful high school students has become serious concerns for educators. 

Alice (http://www.alice.org/) is an easy-to-leam environment which allows users to build 3D visual worlds. 
Instead of creating traditional text-oriented programs which display meaningless messages such as “Hello World” to 
the screen, Alice allows the programmer to create and manipulate interesting objects (such as an ice skater). Those 
objects can be programmed to execute highly visual, exciting actions (such as skate and twirl). Users can create 
interesting environments in a short period of time, thereby increasing their satisfaction and motivation to continue. 
Alice was built by a group of researchers at Carnegie Mellon University under the direction of Randy Pausch 
(Pausch et ah, 1995). The Alice application is freely available and maybe downloaded from the Website 
www.alice.org. The screen shot of the Alice environment is shown in Figure 1. 




Figure 1 The screenshot of the Alice environment. 

As can be seen, Alice provides a set of objects which have been pre-programmed with the basic methods 
for movement and interaction. Students use these as building blocks to create more complex actions, usually to tell a 
story or enact some type of scene. An infinite number of programs can be constructed, from simple interaction to 
elaborate, highly interactive scenes. 
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Need for the Project 



The area served in this project comprises the urban area of east coastal Georgia where approximately 5,600 
families live in poverty. Nearly 4,000 of those families are headed by a single female. While the median household 
income in the county is $42,091, the median income for families with children under 18, headed by single females is 
$26,229. Gotham County Public Schools (GCPS) serves over 44,000 students, over 20% of whom live in poverty. 
Unlike many urban districts, the SCCPS population continues to grow; thus, the need for support of poverty-bound, 
at-risk youth is increasing. 

According to the 2007 National Assessment of Educational Progress (NAEP), only 25% of Georgia’s 8 th 
graders reached the proficiency level or above in mathematics. More students (36%) scored below a basic level. In 
science, Georgia did not participate in the NAEP assessment in 2007; however, in 2005, 75% of Georgia’s 8 th 
graders scored at a basic or below level. Furthermore, in reading barely 26% of Georgia’s 8 th graders scored at a 
proficient or above level since 2002. And, in writing, Georgia students scored below the national average at every 
grade level. As reported by the Georgia Department of Education, Gotham County Schools did not meet AYP 
(Adequate Yearly Progress) in 2008, with more than 25% of the district’s schools reported as not meeting standards. 
Of the district’s seven high schools, only two passed AYP standards. 

The high school completion rates in Gotham County are low, with fewer than 60% of students completing 
high school in four years, while the dropout rates are high. Moreover, the dropout rate for English Language 
Learners in the system is three times the state average, having grown more than 200% over the past three years. 
Finally, the dropout rate for economically disadvantaged students is higher than even that of students with 
disabilities. 

This pilot program project will provide services to a group of students at-risk for becoming statistics in the 
growing numbers of unsuccessful high school students in the urban areas of Gotham County. This will be 
accomplished through the creation of a technology outreach program, in conjunction with Armstrong Atlantic State 
University’s Educational Technology Center, housed in the AASU College of Education in partnership with 
Woodville-Tompkins Technical Career Institute, Gotham County Public School System. 

The proposed project will involve the following: 

• Intervention for at-risk students transitioning from 9 th to 10 th grade 

• Three groups of 20 students per term (60 total) will be identified who meet at-risk characteristics such as: 

o Repeated grade 
o Promotion failure 

o Reading or math test scores below grade level 
o Attendance problems 
o Documented behavioral issues 

• Two-year plan to teach students programming skills using Alice, an innovative 3D programming 

environment that makes it easy to create an animation for 

o telling a story, 
o creating an interactive game, or 
o creating a video to share on the web. 

The Project Design 

“LET TF1E GAMES BEGIN’’ will involve students on a variety of levels. Each group will start with a 
four-day pre -program induction camp prior to the beginning of the school term (in summer and during Christmas 
break). Once the program begins, the students will attend classes twice a month on Saturday. In semester two of the 
program, students from semester one will become mentors to those new in the program. At the end of semester two, 
the original group is ready for an advanced camp, while those who finished the core program in semester two 
become mentors to yet another new group. 

Students will receive a netbook for use during their entire program, making it possible for them to continue 
working on projects during the time when they are not meeting in class. Students will also receive a stipend at the 
end of each term, based upon accomplishment of their targeted goals and exemplary attendance. Finally, students 
will demonstrate their success at the end of each term with presentations to a wider audience. 
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Theoretical Framework 



Researchers found that the causes of school failure are largely attributed to the relationship between the 
social cultural conditions and cognitive conditions. 

1. For at-risk students, especially minority group students, the gap between what learned in school and 
out of school is much wider than that of their peers (Banks, 1988). 

2. At-risk students often fail to develop higher order cognitive skills because they are placed in 
classrooms that deemphasize the need for them (Levin, 1988). 

3. At-risk students are seldom given tasks that require considerable student involvement and decision 
making which are the important forces to lead students to engage in learning (Salomon et. ah, 1991; 
Levine, 1988; Oaks, 1985). 

They found the positive role of educational technology in the three aspects of school failure mentioned 
above. Many researchers contended that the characteristics of the multimedia can help improve at-risk students’ 
cognition capability and deprived social connections to learning. The use of interactive multimedia allows students 
to operate and see phenomena simultaneously in several different symbol systems (graphs, pictures, sounds). This 
can help the students leverage their mental representations to create knowledge structures which help deepen student 
understanding (Kozma & Croninger, 1992). Multimedia such as Alice 3D programming provide students with rich 
mental models of situations, mental models that they would otherwise need to construct on their own with text 
(Fleneghan et al., 1992). The visual nature of the multimedia is more likely to activate the situation-based prior 
knowledge that students already have and connect their new learning to it (Fleneghan et al., 1992). Task of learning 
3D programming involves in rich, interactive simulation that require the need for the basic skills in higher order 
thinking. Instruction within such environments shifts the emphasis from information giving and receiving to an 
emphasis on finding relevant information and learning how to solve problems, ask questions, think critically, and 
communicate ideas (Cole& Griffin, 1987; LCF1C, 1989). 

Project Goals for Participant Learning 

Among our goals for participant learning are the following: 

o Students reach targeted benchmarks (aligned with ISTE standards for students) 
o Improved grades (raising grades in classes to a 2.00 average or higher) 
o Reduced disciplinary problems in and out of school (fewer referrals among program group as 
compared with non-program peers) 

o Improved regular school attendance (fewer unexcused absences in school attendance as compared 
with non-program peers) 

o Graduation rate for completers higher than non-participant peer group 

o Development of a support and mentoring group that includes university faculty, highly qualified 
in-service teachers, and others linked through personal interactions for the duration of the 
program. 

Measurable objectives 

This program will accomplish the following: 

1. Identify, through a systematic method, 20 participants for each of three “Let the Games Begin” groups, 
resulting in 60 participants over two and a half years. 

2. Provide training in technology, specifically programming and development 

3. Provide participants with requisite knowledge and skills that will positively impact success in school areas 

4. Provide on-going support and program continuity following admission to the program. 

Evaluation Plan 



The evaluation for the Armstrong “Let the Games Begin” Program will consist of both a formative and 
summative evaluation. The formative evaluation will be conducted semi-annually to ascertain progress towards 
meeting desired objectives and to identify possible areas of program improvement. A summative evaluation will be 
done at the conclusion of the program by a program evaluator to determine if program goals and objectives have 
ultimately been achieved. The program evaluator will review the data on student achievement, including 
improvements in student grades, school attendance, behavioral objectives, and project attainment, as measured by 
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ISTE standards for students and issue a summative report at the conclusion of the program to address student 
achievement as directly related to the intervention. 

This study seeks answers to the following research questions: 

• What impact does Alice 3D programming workshop have on the at-risk high school students' perceptions of 
self-confidence, attitude toward school and learning, and problem solving skill improvement? 

• What impact does Alice 3D programming workshop have on academic gains, attendance and behavioral 
issues of at-risk high school students? 

• What impact does Alice 3D programming workshop have on at-risk high school students' perceptions of 
gender difference in computing? 

Data Collection 

Three groups of 20 at-risk high school students per term (60 total) will participate in this study. The subjects of 
this study are at-risk high school students in one of Gotham County Public High Schools. Subjects will be recruited 
with help of the high school principal and technology specialist. The school principal and technology specialist 
identify at-risk students based on the characteristics of at-risk students and encourage at-risk students to join the 3 D 
programming workshop. Upon attending the3 D programming workshop, at-risk high school students will be invited 
to participate in this study. Each subject will participate in competing survey questionnaire two times. Each survey 
will take 15-20 minutes so they will contribute total of 30 to 40 minutes of their time to this study. 

Our project also used a qualitative methodology to explore success or failure stories during workshops. This 
included the case study method (Yin, 2002), use of narrative telling (Connelly & Clandinin, 2006), and the 
appreciative inquiry method (Cooperrider & Sorenson, 2005). 

The following questions will be asked in 5 Likert Scale. 

1. 1 hope that my future career will require the use of 3D programming. 

2. 1 like to use programming to solve problems. 

3. 1 have a lot of self-confidence when it comes to computing courses. 

4. 1 do not like using programming skills to solve problems. 

5. 1 am confident that I can solve problems by using computer applications. 

6. The challenge of solving problems using 3D programming appeals to me. 

7. Developing computing skills will be important to my career goals. 

8. 1 would voluntarily take additional computer programming courses if I were given the opportunity. 

9. 1 think computer programming is interesting. 

10. 3D programming improved my problem solving skills in my life. 

1 1 . 1 would like to learn more about careers related to computer programming. 

12. 1 have become more interested in school since learning 3D programming. 

13. 3D programming improved my self confidence. 

14. 3D programming club improved my computer skills. 

The additional 3 open-ended questions will be asked. 

1. Describe, in detail, what programming means to you. 

2. Please describe the characteristics of a person with a career in programming. 

3. In your opinion, describe how programming can help you to solve problems in your lives? 

Data Collection 

The subjects will be invited to complete a survey questionnaire right before the Alice 3 D 
programming workshop. This research is a pre-experimental and quantitative study of one group pre-test and 
post-test model. They will be participating in the pretest which will be administered right before the 
participation of the Alice 3 D programming workshop. After completion of the Alice 3 D programming 
workshop, they will participate in the post-test conducted. They will also be asked to participate in an interview. 

For the reliability test of the instrument, they will be asked to complete the same questionnaire three weeks after 
the first completion of the survey questionnaire. The semi-structured interview will be conducted to further 
investigate the research questions. The primary investigator will observe the workshop site to explore and 
reflect upon the progress of the project. 
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Conclusion 



The article outlined AT & T-sponsored educational research project and the evaluation plan of the 3D 
visual programming use in the project. Current evidence suggests that emerging technologies can facilitate learning, 
including the learning of student at risk of school failure. We are enthusiastic about these new technologies, but at 
the same time we are careful about the approaches to and infrastructure of technology projects. This research project 
will offer suggestions for educators who would like to integrate technology in improvement of at-risk students. 



References 



Banks, J. A. (1988). Ethnicity, class, cognitive, and motivation styles: Research and teaching implication. Journal of 
Negro Education, 57, 452-466. 

Connelly, F. M., & Clandinin, D. J. (2006). Narrative inquiry. In J. L. Green, G. C. Camilli, & P. B. Elmore (Eds.), 
Handbook of complementary’ methods in education research (pp. 477-488). Washington, DC: AERA. 

Cole, M., & Griffin, P. (Eds.) (1987). Improving science and mathematics education for minorities and women. 
Madison: University of Wisconsin, Wisconsin Center for Educational Research. 

Johnson, G. M. (1998). Principles of Instruction for At-Risk Learner. Preventing School Failure, 42 no4 167-74. 
Cooperrider, D. L., & Sorenson, P. (2005). Appreciative inquiry’: Foundations in positive organization development. 
Chicago, IL: Stipes. 

Georgia Department of Education (2011). Reporting Adequate Yearly Progress as required under the No Child Left 
Behind Act of 2001. Retrieved February 10, 

2011 from http://www.doe.kl2.ga.us/datareporting.aspx 

Kozma, R. B. (1991). Learning with media. Review of Educational Research,61(2), 179-211. 

Laboratory of Comparative Human Cognition (LCHC). (1989). Kids and computers: A positive vision of the future. 
Harvard Educational Review, 59, 61-73. 

Merriam, S. B. (1998). Qualitative research and case study research. IEEE Transactions on Professional 
Communication, 40(3), 182-196. 

Pausch, R., Burnett, T., Capeheart, A. C., Conway, M., Cosgove, D., Deline, R., Durbin, J., Gossweiler, R., Koga, 

S., & White, J. (1995). Alice: Rapid Prototyping system for virtual reality. IEEE Computer Graphics and 
Applications. 

U. S. Department of Education National Center for Educational Statistics (2011). National Assessment of 
Educational Progress: The nation ’s report card. Retrieved February 10, 

2011 from http://nces.ed.gov/nationsreportcard/states/ 

Yin, R. (2002). Case study research: Design and methods (3rd ed.). Thousand Oaks, CA: Sage. 



393 




Mobile Computing: Perspectives on Design, Learning, and Development 



Susan M. Land, Heather T. Zimmerman, Orrin T. Murray, Simon Hooper, 

Priya Sharnia, James Mundie 

Penn State University 

The relationship between technology and its use in education has provoked debate for decades. Some 
Instructional Designers have argued that technologies should be viewed as tools for content delivery and that the 
expected educational benefits are limited to efficiency gains. However, others believe that technology can be used to 
facilitate knowledge creation. In other words, new technologies can be used to teach in ways that are radically 
different from the past and that such approaches have the potential to transform what it means to teach and learn. 

Recently, educational technologists have begun to explore the potential for mobile technologies. For each 
of the last four years (i.e. 2008-2011 ) the Horizon Report has highlighted Mobile Computing as one of two 
emerging technologies likely to enter into mainstream use during the next five years (Johnson, et al., 2011). Despite 
the optimism, little formal research has been conducted into how mobile technologies are being used in classrooms. 
An examination of current educational software for mobile technologies (i.e. apps) at various educational and 
business sites (e.g. iTunes store; http://www.accreditedonlinecolleges.com/blog/2009/10Q-most-educational-iphone- 
apps ; http://www.studvbyapp.com ) suggests little difference between app design and traditional educational 
software. Most educational apps fall into three categories: audio/ video lectures, drill and practice, and flashcards. 
Apparently, designers either ignore, or are unaware of, differences between traditional and mobile technologies. 
When new technology is used simply to replicate current educational practice, changes to teaching and learning 
should not be expected. 

Understanding how mobile technology can support teaching and learning involves developing insight into 
what differentiates technologies. Some differences between mobile technology and traditional computers are 
obvious: mobile computers are generally light and easily transportable, displays are small and cannot easily render 
large images, and they include touchscreen capabilities. However, other differences are less apparent. For example, 
mobile technologies often incorporate state-of-the-art hardware capabilities that enable different forms of activity 
between users, or between a user and the device. Likewise, the combination of high device portability combined 
with improved device networking capabilities has led to essentially ubiquitous internet access. Although not 
necessarily a technological change, form differences make the devices appropriate for use in settings not usually 
associated with computer use such as on crowded trains. For the designer, knowing and understanding the range of 
differences is essential to effective design. 

Donald Norman used the term affordance to refer to the physical clues emanating from an object that allude 
to its function: "...the term affordance refers to the perceived and actual properties of the thing, primarily those 
fundamental properties that determine just how the thing could possibly be used" (Norman, 1988, p.9). Although, 
strictly speaking, the term affordance refers to a device’s physical properties or capabilities, here we extend 
Norman’s definition to include other features that impact the designer. Mobile technologies can be compared with 
more traditional computers using a framework that includes three dimensions: 

• The properties of a device 

• The infrastructure that impacts its use 

• The social and contextual factors that influence how and where technology is used 
Device Properties 

Designers must be aware of how the properties (or features) of a device affect its design. Device properties 
can be classified into three categories: technological, designed, and form factor. 

Technological properties are the hardware capabilities built into a device. In addition to the basic hardware 
included in all computers (i.e. a display device, CPU, RAM, and storage medium) mobile technologies include 
selections from a range of physical technologies such as an accelerometer and motion sensor, a microphone and one 
or more digital cameras, GPS and compass, Bluetooth capability, touch screen, etc. 

Designed properties are the set of capabilities made possible through the design of software that allows 
users to interact with technological properties of the device. Mobile technology enables the design of new 
interaction patterns that establish different forms of interactivity and navigation . 
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Form factor properties are other physical capabilities (e.g. size, color, shape, hardness, weight, etc.) that 
affect how, when, and where people choose to use a device. Form factors also provide clues as to their function and 
operation (Norman, 1988). Gibson argued that in addition to learning the physical and designed capabilities that 
exist in objects, humans recognize or understand the potential capabilities of an object: Objects have demand 
characteristics. These physiognomic qualities (a term derived from the noun physiognomy meaning the art of 
judging human character from facial features) refer to the set of values or emotions that people project onto objects. 
For example, Gibson (quoting Kofka) noted that thunder evokes fear, water induces drinking, and fruit suggests 
eating. 

Similarly, mobile devices create behavior expectations from their users: tablet PCs (and most definitely 
smart phones) are designed to encourage holding; most mobile technologies are touch sensitive— they invite 
touching. Thus, a smart phone, which is easy to hold, is more likely than a laptop to be used while commuting. 
Portability and ubiquitous network connectivity make the device always accessible and ready for use as a 
communication tool, enabling not only the capture of photographs/ video, but also posting the images online or 
catching up on email/ RSS feeds. 

Device properties create new design potential and limitations. One such example concerns how designers 
must adapt to the presence of a touch screen. A touch screens changes interaction patterns by allowing a user to 
interact and operate the device with a single finger, thereby creating the illusion of interacting directly and naturally 
with screen content. When designed appropriately, touch screens may improve the ability to use computing devices 
by removing a layer of abstraction present in typical input devices (i.e. mouse, keyboard, etc). 

Though touch screens can be simple and intuitive to use (even by children as young as 2 years old) and 
have the potential to promote complex interface task-completion, important limitations are associated with touch- 
screen use (Benko & Wigdor, 2010). Touch screens tend to increase location ambiguity (lower precision and higher 
accidental activation) and are associated with poor feedback resulting reduced system reliability and increased user 
frustration. 

Environmental Infrastructure 

Usage of handheld technology is enabled by the features of the device space. The device space refers to the 
environment established to support the physical technology. Without an established device space the user is isolated 
and unable to communicate with other people or devices. For example, desktop computers in the early 1980s were 
isolated as data could be transported only via tape or floppy disk, and connectivity was generally limited to printed 
documents. 

The most apparent manifestation of the device space involves the infrastructure required to provide 
network connectivity. Whereas desktop and laptop computers are generally connected via Ethernet or wireless 
Internet connections, mobile users rely on the availability of mobile broadband signals either from wireless networks 
or cellular access. More sophisticated networks (e.g. 3G vs. 4G) enable faster connection speeds which, in turn, 
create more transparent network connections for end users. 

The functionality of cellular networks throughout much of the world is differentiated according to carrier 
(e.g. Verizon, AT&T, etc.) with each carrier deciding how the network is implemented in the device space. This 
imposes additional factors which must be considered when designing applications for educational use, including 
cost, network availability, platform availability, market share, and brand perception. For example, if an app requires 
large amounts of bandwidth to use, as when streaming video, it may be prohibitively expensive for some users, or 
simply unavailable in certain areas due to lack of network capacity. Additionally, some networks may lack certain 
functionality such as the ability to tether a smartphone to a laptop or the ability to receive data simultaneously with 
voice calls. Mobile device market share and brand perception play a role in determining what devices are available 
to consumers in a given area, relative numbers of users for each platform, and what features exist on those devices. 
Additionally, different manufacturers create devices that may look similar but have vastly different device properties 
and different user interface conventions. 

The next few years are likely to see considerable growth in infrastructure such as cloud computing creating 
access to large data storage space. As this infrastructure matures, users will become more able to use mobile devices 
in increasingly sophisticated ways to not only access content from educational providers, but also to interact with 
peers and produce content for others to consume. The properties and capabilities of mobile devices, combined with 
emergent social networks and mobile social media tools in the context of a robust and powerful infrastructure will 
create new opportunities for teaching and learning while simultaneously blurring the lines between the devices 
themselves. 
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Social, Cultural, and Contextual Factors 

The manner in which people use mobile technologies is affected by various social phenomena that appear 
to evolve rapidly. Although the direction of social developments is difficult to predict, designers would do well to 
understand their implications and to respond accordingly. Below we describe three such trends that have immediate 
implications for designers. 

Differentiated technology use 

Important design implications are associated users interact with different technologies. Users differentiate 
computing tasks: they select technologies to match the context and the nature of the activity. Desktop computers 
(and perhaps even laptops) may become the ‘production workhorse’ given its massive processing capability, data 
storage, and viewing capabilities. In contrast, smartphones are likely to be used for small tasks that require limited 
information and processing. For this reason, smartphones may be better suited to informal education spaces than for 
current practice in formal education. 

The App Paradigm 

Diverse social factors affect users’ decisions concerning whether to use app software. Referred to as the 
‘App Paradigm’ these factors affect how users interact with software on their devices. The App paradigm changes 
how users purchase software and how publishers/ marketers provide content. Whereas the web browser served as a 
general software environment, the app culture is highly specialized and personalized content. Compared to the 
desktop or files-and-folders paradigm, the app paradigm is a more direct and secure channel between publisher and 
consumer of content. The app paradigm allows publishers of content to interact directly with customers, providing a 
la carte content on-demand either all at once or in arbitrary chunks. 

Market Fragmentation 

Both the smartphone and tablet markets remain fragmented and future market-trends remain difficult to 
predict (Blodget, 2011). For example, as of spring 2011, Android held 33% of the smart phone market compared to 
29% for BlackBerry, and 25% for Apple. Market fragmentation makes designing for specific audiences relatively 
difficult, as design tends to be platform specific. In contrast, the tablet market appears to be more stable. Brown 
(2011) predicted that the current market leader (i.e. Apple) is likely to maintain its majority share of the tablet 
market until at least 2015. 

Mobile devices vary both in physical dimension and in operating systems which, in turn, have implications 
for development specifications and the programming languages used for development. Moreover, the programming 
languages available for IOS (i.e. iPad/ iPhone), Windows7 and Android operating systems vary greatly in 
sophistication with the result that designers often select a development platform to minimize development time. 

Market fragmentation causes designers to attempt to predict which devices will survive (or even dominate) 
in order to maximize exposure to their applications. It is more feasible to translate existing content in the formal 
education space directly into something that works on a tablet, because the form factor is characterized by greater 
screen real estate approaching that of a regular computer. The slate or tablet form factor bridges the gap between 
laptop/desktop and smartphone, with a larger size allowing more powerful computer components for a potentially 
more powerfid multimedia experience. 

Social and Contextual Factors 

From smart phones to tablets, the current growth rate of mobile computing devices in the hands of people is 
breathtaking. This fall Apple and Google announced that between them upwards of 400 million mobile computing 
devices have been sold-in a little more than six years. And, while individuals might be fueling the bulk of these new 
purchases education institutions are not very far behind. Much of the motivation, among educators for these devices 
seems to be drawing on the promise of 1 : 1 computing that drove Governors Angus King or Ed Rendell to launch 
state-wide 1:1 initiatives in Maine and Pennsylvania. While numerous K-12 school district and universities have put 
in place 1:1 initiatives, the price of a computer has always been an issue. At the prices below $500 many of the cost 
concerns of schools are melting away. 
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At the core of education 1 : 1 computing initiatives is a sense that what students have traditionally done in 
schools has not connected well with what people do outside of schools. In making the case for new theories of 
learning, Bransford (1999 pp. 73-78) and his colleagues took up some of this issue, and it is a central tenet of the 
most recent education reform wave. From Science to English, state curricular standards have been revised, over the 
last decade, to emphasis and build skills reflecting the type of work or problem solving that happens outside of 
school. Moreover organizations like P21 (see, P21.org) and economists like Levy and Murnane (2004) have been 
arguing for the last decade that the world in which high school students will graduate into is demanding skills our 
schools are not developing. 

One of the foundations of contemporary learning theories rests on the notion that learning and how we 
come to know is contextually and socially dependent. In particular, the social aspects of these theories imply, 
collaboration. In the example Brandford and colleagues use to promote their vision of how learning should more 
easily transfer to activities people do outside of schools, collaboration is contextualized as problem solving among a 
group of people on a particular issue (e.g., 1999, p. 74). It is somewhat surprising then, that in the midst of this 
explosion of mobile devices, the use of these new devices seem to support the atomization of individuals. To sure, 
mobile devices like smart phones and tablets (e.g., Apple’s iPad) have connective capabilities, which can support 
“social" connections. However, a recent study of application available for iOS devices suggests that many are 
designed to support individual consumption and or creation of content (Murray & Olcese, in press). Moreover, 
applications specifically targeted towards the education space generally support learning through behavioral 
mechanisms. These results stand in stark contrast to education reform efforts that are attempting to address the 
apparent disconnect between what goes on in schools and what goes on outside of schools. 

As has been noted earlier, it is not the case that device manufactures or operating systems designers have 
not built in capabilities to support collaboration. For example, a game that runs on iPads like Scrabble illustrates the 
potential to use various networking capabilities that can support real-time collaboration, or gaming, synchronously. 
Applications like Scrabble, however are the rare exception, across all categories of applications. One need only look 
at the various sites that track application downloads (e.g., 148apps.com) to see that the bulk of apps downloaded for 
mobile devices are focused on consuming content. 

Given the number of applications that have been created and downloaded (15 billion for Apple’s iOS 
devices over the last three and a half years) in such a short period of time, it would appear that one issue is not that 
application development in this space is either especially difficult, or that applications are not interesting to users. 
Rather, the issue is that the field of instructional design has not done an adequate job of offering principles to help 
guide the development of applications that explicitly support collaboration. Designers of application that are targeted 
at education need to understand what collaboration is, and how to design it into their applications for mobile 
computing devices. Of the many challenges related to developing these principles an important first one would be 
defining what is meant by collaboration. Is collaboration something that must have a minimum number of people? 
What are the spacial and temporal parameters associated with collaboration? Moreover, how is collaboration in non- 
school settings changing in the face of innovation? 

Without clear principles for developing applications that encourage collaboration-and provide material 
support to contemporary theories of learning-it is likely that the impact of mobile devices in education is headed for 
the same fate of the many educational technologies that came, full of promise, before. 

Exploring Individual Capture and Use of Data with Mobile Devices 

The Pew Internet organization’s mobile access report suggests that 59% of US adults access the internet 
using mobile devices and 38% have uploaded data to the internet using mobile devices (Smith, 2010). While much 
of this data may be uploaded to social sites such as Facebook, Flickr, and Twitter, there is also an increase in 
uploading content that serves to log one’s actions or thoughts (Wolf, 2010). Content that is uploaded can vary in 
focus, ranging from mood status and time spent on tasks, to precise run times and speed, or blood sugar numbers, 
which are based on more precise and objective data. Mobile devices, especially mobile phones, are becoming 
smaller, more multifunctional, and more capable of integrating diverse features including multiple types of data 
uploads. Correspondingly, there are a growing number of mobile apps that allow one to record and revisit data about 
personal thoughts, actions, and physiological and affective data (to name a few). Wolf (2010) terms this trend as a 
move to ‘quantify oneself and suggests anecdotal evidence that such detailed, objective records of actions and data 
can significantly affect how an individual perceives, reflects on, and subsequently modifies behavior or practice. 

Current research and interests in mobile enhanced learning or mlearning have tended to focus on 
applications, or apps as they are more commonly known, that are add-ons to existing learning management systems 
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(LMS) to support formal learning situations. This may include the ability to get course updates via mobile devices, 
and engage in some types of communication via the mobile LMS interface. In other instances, apps have been 
developed to help students record and interpret location based data to serve formal learning goals related to 
mathematics (Roschelle et al., 2010; Zurita and Nussbaum, 2007) and science (Breuer, Konow, Baloian, & Zurita, 
2005). Park (2010) classifies a variety of such learning apps under the umbrella of transactional distance and 
identifies how these apps support different levels of dialogue, structure, and autonomy in enhancing distance 
learning. Such a classification is useful for understanding how to enhance formal learning with mobile apps, 
however, there exists an important opportunity for looking at how individuals use mobile apps to support self- 
initiated learning and interest driven activities. 

Interest-driven and self-initiated use of mobile apps 

Interest driven learning and learning by participation in activities within a community accounts for a large 
proportion of life-long learning outside of formal environments (Bell, Lewenstein, Shouse, & Feder, 2009). With the 
growth of online networks and communities that exist to serve different needs, more individuals are assuming 
responsibility for building knowledge and skills in a given area of interest. Topics of interest can vary from cooking, 
exercise, and running, to managing a disease and weight. The website, quantifiedself.com, lists approximately 160 
apps that allow users to upload their data to a chosen site. For example, Islet™ allows users to record blood glucose 
levels, insulin, and food data and then view graphs and charts of the data, or export the data via email to professional 
advisors, including doctors or nurses. Among the multiple applications for tracking exercise patterns (running, 
biking, etc.,) Runtastic™ allows users to record data from each exercise session including speed, calories burned, 
pace, heart rate, and route. These data can be tracked and stored online. 

In some cases, the apps and data storage are accompanied by access to a community of similar users. For 
example, SparkPeople™ has communities for nutrition, wellness, fitness, and motivation, and along with the 
possibility of uploading and tracking data, users also have the option to consult on topics of interest with experts or 
other amateur community members like themselves. Research conducted by Hwang et al. (2007) on the weight loss 
discussion board within this network indicates that the majority of advice provided was neither erroneous nor 
harmful, but that there was more likely to be erroneous information when dealing with topics such as medication. 
Similarly, Funk et al. (2010) evaluated the use of a website for long term weight maintenance associated with adults 
at risk for cardiovascular disease. They found that consistent users of the site (with more logins, more number of 
minutes spent on the site, more exercise entries, and increased used of website features) were more successful that 
sporadic users in maintaining long term weight loss. 

While a range of scenarios are possible within this area, what is important to note is that people are 
increasingly willing to document personal data as well as data about specific practices/routines. We can speculate 
that these data prove useful in charting trends in their behaviors or states, which can become the focus of individual 
learning and modification of practice (e.g., Wolf, 2010). As also noted, when these tools are embedded within a 
supportive, community-based framework, that includes some type of expert guidance, there is increased likelihood 
of success in changing and maintaining practices. As a first step, we should explore the range and types of individual 
use of and engagement with mobile apps, such that it can inform integration of such type of individually-led 
engagement with formal learning environments. A second step might include designing specific apps that are 
educationally focused, that support the types of interests and engagement valued by individuals. 

One Implementation: Augmented Reality as Mobile Learning 

Mobile computing devices are marked by physical characteristics of size and portability. As such, they can 
be used to support learning across multiple settings, both within and outside of formal classrooms (Pea & 
Maldonado, 2006). Given the portability of mobile devices, students are no longer physically tied to a desktop 
computer as the only means to engage in computationally-mediated learning activities. Rather, students can explore 
their everyday surroundings using portable, wireless handheld devices, thus extending the opportunities for learning 
to contexts and/or locations that normally lie beyond the boundaries of the school classroom. This connection to 
everyday surroundings creates potential opportunities to anchor learning in situated, authentic and everyday contexts 
(Bransford, et al., 2000). 

One form of mobile computing that takes advantage of the physical characteristics of place is augmented 
reality (AR). Augmented reality is a technology-enhanced environment that combines elements of a real-world 
physical space with virtual material (e.g., text information, videos, audio, gaming scenarios) at the same time, and in 
real time (Rogers et al., 2004). Popular augmented reality environments are downloadable commercially as apps 
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onto a mobile device. For instance, one popular commercial app is StarWalk, which enables users to point their 
mobile devices to a particular location in the night sky and see real-time images of constellations, planets, and stars 
overlaid onto their device screen. 

The 2011 Horizon Report (Johnson et ah, 2011) suggests that augmented reality offers “strong potential to 
provide both powerful contextual, in situ learning experiences and serendipitous exploration and discovery of the 
connected nature of information in the real world.” (p. 17). Klopfer (2008) refers to the extension of augmented 
reality to learning environments as augmented learning. Augmented learning environments rely on context- or 
place-sensitive information to provide a more immersive learning experience. Students carry mobile, wireless 
devices through a real world context and “engage with virtual information superimposed on physical landscapes 
(such as a tree describing its botanical characteristics or a historic photograph offering a contrast with the present 
scene).” (Dunleavy, Dede, & Mitchell, 2009, p. 8). Augmented learning environments may utilize commercial AR 
apps, but they are designed specifically with goals to shape learning on demand in a contextually-rich context. 

One of the projects affiliated with our Mobile Computing Research Group at Penn State involves the 
design, implementation, and research of an augmented learning environment ( Tree Investigators) at the University 
Arboretum. Our group began the Tree Investigators project as an opportunity to engage in outreach with the PSU 
Arboretum, specifically an Arbor Day field trip event for 300 4th graders from 3 local schools. The Arboretum has 
17,000 individual plants that represent over 700 different species on a 35 acre plot of land. We staffed a station at 
the event for children to use iPods and iPads to explore the tree and plant science within the arboretum. The idea 
behind Tree Investigators is to tailor content to a fourth grade level, to PA learning benchmarks, and to the scientific 
value of the Arboretum's collection. The augmentation provided learners with additional resources to engage their 
physical space. The technology only added information to the physical place; it was not a stand-alone program. In 
this way, augmented learning has a strong reliance on place-sensitive aspects from the original learning setting. The 
blend of the technology augmentations and key elements from the outdoor learning space changes the activity by 
combining aspects from two settings (technological and physical) into one. 

We augmented student learning using the technology in two main ways. First, we augmented students’ 
physical observations of eight different trees with text and images accessed via the QR (Quick Response) codes that 
were indexed to specific trees to highlight variations in physical characteristics. We used Microsoft™ Tag Reader to 
bring specific web-based content to an iPod or iPad that was uniquely relevant to the site being explored. The Tree 
Investigators app allowed youth to explore trees onsite at the Arboretum, and when the children used the camera on 
an iPad or iPod to snap a picture of a “tag” made with Microsoft™ Tag Reader, additional photographs and textual 
information (written at a reading level of elementary aged students) appear on their touch screen as a customized 
website (see figure 1): 

Figure 1: An example QR Barcode Tag 




This information, developed and designed by the research team included three main characteristics of the tree being 
investigated: 1) leaves/needles, 2) fruit elements, and 3) bark features (see figure 2). Where possible, we augmented 
images and information that were not directly perceptible to the students at that time (e.g., fall leaf colorings; spring 
flowering). 
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Figure 2: Tree Characteristics that were Augmented via MS Tag Reader 



AT&T 3G 10:11 AM 



You can look carefully at a tree to find out more. 
Through detailed observations, you can find 
information that will help you identify a tree. The 3 
best ways to identify a tree arc below. Touch each 
button to learn more about this wonderful tree. 





Second, we augmented the learning space by creating a fictitious “tree mystery” scenario as a context for 
applying observations about trees. Students were told that they were going to help figure out the species of the 
mystery tree that was donated to the Arboretum. Students received clues to figuring out the mystery tree via large 
AR markers that displayed an image of one of the characteristics of the mystery tree. When the AR markers were 
held in front of the webcam, the content associated with that marker (i.e., a unique characteristic or feature of the 
mystery tree) was displayed on the computer screen (see figure 3). Students used a commercially-made US Trees 
app to help narrow their search by answering dichotomous key questions. 

Figure 3: AR Markers Displaying Mystery Tree Characteristics 




In sum, the goal of our research group is to apply learning sciences and constructivist learning theories to the 
development of augmented learning apps for mobile devices (like iPods, iPads, smart phones, tablets). We aim to 
develop more robust learning theories that reflect the way that youth learn across settings (e.g., across school, home, 
extra-curricular activities) with technologies. We want to develop more innovative, mobile educational technologies 
that extend educational computing beyond school desktops and out into the homes, lives, and experiences of youth. 
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Abstract: There are many different ways for people to be educated and trained in regards 
to specific information and skills they need. These methods include classroom lectures 
with textbooks, computers, handheld devices, and other electronic appliances. The choice 
of learning innovation is dependent on individual’s access to various technologies and the 
infrastructure environment of the surrounding community available. In a rapidly 
changing society where there is a great deal of available information and knowledge, 
adopting and applying information at the right time and right place is needed to main 
efficiency in both school and business settings. Augmented Reality (AR) is one 
technology that dramatically shifts the location and timing of learning and training. This 
literature review research describes Augmented Reality (AR), how it applies to learning 
and training, and the potential impact on the future of education. 



The general meaning of AR 

Augmented Reality (AR) is a technology that allows computer-generated virtual imagery information to be 
overlaid onto a live direct or indirect real-world environment in real time (Azuma, 1997; Zhou, Duh, & Billinghurst, 
2008). AR is different from Virtual Reality (VR) in that in VR people are expected to experience a computer- 
generated virtual environment. In AR, the environment is real, but extended with information and imagery from the 
system. In other words, AR bridges the gap between the real and the virtual in a seamless way (Chang, Morreale, & 
Medicherla, 2010). 

The origin of AR in learning and training. 

According to Johnson, Levine, Smith, & Stone (2010), the history of AR goes back to the 1960s and the 
first system was used for both Augmented Reality and Virtual Reality as well. It used an optical see-through head- 
mounted display that was tracked by one of two different methods: a mechanical tracker and an ultrasonic tracker. 
Due to the limited processing power of computers at that time, only very simple wireframe drawings could be 
displayed in real time (Sutherland, 1968). Since then. Augmented Reality has been put to use by a number of major 
companies for visualization, training, and other purposes. The term ‘Augmented Reality’ is attributed to former 
Boeing researcher Tom Caudell, who is believed to have coined the term in 1990. 

Marker- and Markerless-based AR. 

According to Johnson, et al. (2010), augmented reality systems can either be marker-based or markerless- 
based. Marker-based applications are comprised of three basic components which include a booklet for offering 
marker information, a gripper for getting information from the booklet and converting it to another type of data, and 
a cube for augmenting information into 3D-rendered information on a screen. On the other hand, markerless-based 
applications need a tracking system that involves GPS (Global Positioning System), a compass, and an image 
recognition device instead of the three elements of maker-based systems. Markerless applications have wider 
applicability because they function anywhere without the need for special labeling or supplemental reference points. 

Adopting AR in learning and training. 

According to Chang, Morreale, and Medicherla (2010), several researchers have suggested that students 
and trainees can strengthen their motivation for learning and enhance their educational realism-based practices with 
virtual and augmented reality. In spite of a great amount of research during the last two decades, adopting AR in 
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learning and training is still quite challenging because of issues with its integration with traditional learning methods, 
costs for the development and maintenance of the AR system, and general resistance to new technologies. Now that 
AR, however, has the promise to attract and inspire learners with exploring and controlling materials from a 
diversity of different perspectives that have not been taken into consideration in real life, AR in education and 
training is believed to have a more streamlined approach that has wider user adoption than ever before due to the 
improvement in computer and information technology. Kerawalla, et al. (2006) stated that even though many AR 
applications have been developed for educational and training purposes since the advent of AR in the late 1960s, its 
potential and pragmatic employment has just begun to be explored and utilized in real life. He emphasized that AR 
has the potential to have learners more engaged and motivated in discovering resources and applying them to the 
real world from a variety of diverse perspectives that have never been implemented in the real world. 



How it applies to learning and training 

Johnson, et. al. (2010) stated, “AR has strong potential to provide both powerful contextual, on-site 
learning experiences and serendipitous exploration and discovery of the connected nature of information in the real 
world.” (p. 21). AR has been experimentally applied to both school and business environments, although not as 
much as classic methods of learning and training during the last two decades. In addition to that, now that the 
technologies that make augmented reality possible are much more powerful than ever before and compact enough to 
deliver AR experiences to not only corporate settings but also academic venues through personal computers and 
mobile devices, several educational approaches with AR technology are more feasible. Also, wireless mobile 
devices, such as smart phones, tablet PCs, and other electronic innovations, are increasingly ushering this 
technology, AR, into the mobile space where the AR applications offer a great deal of promise, especially in 
learning and training. 

AR in school 

Professionals and researchers have striven to apply AR to classroom-based learning within subjects like 
chemistry, mathematics, biology, physics, astronomy, and other K-12 education or higher, and to adopt it into 
augmented books and student guides. However, Shelton (2002) estimated that AR has not been much adopted into 
academic settings due to little financial support from the government and lack of the awareness of needs for AR in 
academic settings. 

AR in business 

In corporate venues, AR is a collaborative, skill-learning, explainable, and guidable tool for workers, 
managers, and customers. Also businesses have a better environment than those of educational settings regarding the 
ability to maintain the costs and support of AR applications. Many corporations are interested in employing AR for 
the design and the recognition of their products’ physical parts. According to the evaluation of Shelton (2002), for 
example, enterprises not only may imagine designing a car in three dimensions in which they can make immediate 
changes when needed but also can create virtual comments that explain to the technicians what needs to be fixed. 



The current position of AR in learning and training 

During the last few decades, many professionals and researchers have been developing pragmatic theories 
and applications for the adoption of AR into both academic and corporate settings. By virtue of those studies, some 
innovations of AR have been developed and are being used to enhance the learning and training efficiency of 
students and employees. In addition to that, there are a great number of studies going on to improve the 
compatibility and applicability of AR into its real life. However, according to Shelton & Hedley (2004), many 
questions still linger about its use in education and training including issues of cost effectiveness, of efficiency 
between AR instructional systems and conventional methods, and the like. 
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Augmented astronomy 

In an astronomy class, students learn about the relationship between the earth and the sun. For the sake of 
students understanding, educators may employ AR technology with 3D rendered earth and sun shapes. 

Shelton’s (2004) study described the following: 

The virtual sun and earth are manipulated on a small hand-held platform that changes its orientation in 
coordination with the viewing perspective of the student. The student controls the angle of viewing in order to understand 
how unseen elements work in conjunction with those that were previously seen. (p. 324) 




Figure 1. A view of a student interacting with real objects (foam core card, table, wall) and artificial 
objects (Sun, Earth, annotations) through the augmented reality interface. This view is as would be seen if 
wearing an HMD (Shelton, 2004). 

As another example for the employment of AR in astronomy, Johnson, et. al. (2010) described Google’s 
SkyMap as an application using AR technology. SkyMap overlays information about the stars and the constellations as 
users browse the sky with the see-through view from the camera on their smart phones, (p. 23) 




Figure 2. Google SkyMap (Retrieved from http://www.voutube.com/watch?v=r>6znyx0gib4 ). 

Augmented chemistry 

Augmented chemistry is an interactive educational workbench that can show students how and what an atom 
or a molecule consists of via AR. Three elements, a booklet, a gripper, and a cube, are required to implement this task 
with both hands. Fjeld & Voegtli (2002) said that the booklet displays components by a printed picture and a name. 
One hand browses the booklet with a gripper which has a button used to connect an atom to the molecular model. 
According to Fjeld & Voegtli (2002), firstly, users bring the gripper around the element in the booklet and get 
information about the element by clicking the button of the gripper. Second, users move the gripper next to a cube, 
called a platform, which holds a molecule. Subsequently, by rotating a cube operated by the other hand, users can 
determine where and how the element connects to the molecule. 
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Figure 3. a) Booklet offering one element per page - here Na - sodium. Each element is represented by a 
pattern, b) Gripper with a button (red) and a pattern, c) Cube with one distinct pattern for each surface 
(Fjeld & Voegtle, 2002). 




Figure 4. System set-up with a typical situation of use: charging the Gripper with an element from the 
booklet (left). The platform (right) holds an unsaturated atom, with which a binding with the charged atom 
may be triggered (Fjeld & Voegtle, 2002). 

Augmented biology 

AR can be used to study the anatomy and structure of the body in biology. The Specialist Schools and 
Academies Trust (SSAT) demonstrated that teachers could use AR technology to show what organs of human 
beings consist of and how they look by watching 3D computer-generated models in the real classrooms. Moreover, 
students may be able to study humans’ organs independently with their camera-embedded laptops and AR markers 
that connect PCs with AR information about biological structures of the human body. (Retrieved from 
https://www.ssatrust.org.uk/achievement/friture/Pages/AugmentedReality.aspx) 




Figure 5. A model of human beings’ internal organs with AR technology that can be used in Biology class 
(Retrieved from http://www.learnar.org). 
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Mathematics and geometry education 

With AR technology, teachers and students can collaborate by interacting with each other for some issues 
on shapes or arrangements. According to Chang, Morreale, & Medicherla (2010), an AR application, called 
Construct3D, specifically was designed for mathematics and geometry education with three-dimensional geometric 
construction models (as cited in Kaufmann, 2006; Kaufmann & Schmalstieg, 2002; Kaufmann, Schmalstieg, & 
Wagner, 2000). This application allows multiple users, such as teachers and students, to share a virtual space 
collaboratively to construct geometric shapes by wearing head mounted displays that enable users to overlay 
computer-generated images onto the real world. 




Figure 6. Students working with construct3D inscribe a sphere in a cone (Kaufmann & Schmalstieg, 2002). 

Furthermore, Kaufmann (2009) determined that AR can be used in dynamic differential geometry 
education in a wide range of ways. For instance, using the AR application, teachers and students can intuitively 
explore properties of interesting curves, surfaces, and other geometric shapes. 

AR in K-12 education 

Freitas & Campos (2008) developed SMART (System of augmented reality for teaching) that is an 
educational system using AR technology. This system uses AR for teaching 2nd grade-level concepts, such as the 
means of transportation and types of animals. This system superimposes three dimensional models and prototypes, 
such as a car, truck, and airplane, on the real time video feed shown to the whole class. Because most children spend 
a great deal of time playing digital games, game-based learning is one way to engage children in learning. Several 
experiments by Freitas & Campos (2008) were performed with 54 students in three different schools in Portugal. 

The results of a number of studies by Freitas & Campos (2008) indicated that SMART helps increase motivation 
among students, and it has a positive impact on the learning experiences of these students, especially among the less 
academically successful students. 

How AR is applied to the business training 

Cultural heritage 

From cultural and traditional perspectives, AR can be used as an influentially interactive tool in cultural heritage 
sites by showing visitors the original images of the sites and informing travelers of historical episodes of the places 
with 3D effects. Vlahakis, et al, (2002) demonstrated on their research of ARCHEOGUIDE (Augmented Reality- 
based Cultural Heritage On-site GUIDE) that the AR tour assistant system provides on-site help and Augmented 
Reality reconstructions of ancient ruins, based on users’ position and orientation in the cultural site, and real time 
image rendering. ARCHEOGUIDE is based on computer and mobile technologies including AR, 3D-visualization, 
mobile computing, and multi-modal interaction techniques. The equipment consists of a Head-Mounted Display 
(HMD), an earphone and a mobile computing unit. But other versions include a PDA or a lightweight portable 
computer with a simple input device. With these AR devices, individuals can visit historic sites and tour around, 
both comparing an original image to an augmented modeling and viewing three-dimensional models of what the 
construction was, how it looked, and who the person was, even though it does not exist any more or just remains 
some ruins. 
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Figure 7. a) The original image of a heritage site, b) An image of AR modeling. Examples of 
ARCHEOGUIDE’s original image and AR modeling (Vlahakis, 2002). 

Industrial maintenance 

In the field of industrial maintenance, AR is a very practical assistance for staff in their highly demanding 
technical work. Henderson & Feiner (2009) observed that corporate sectors such as military, manufacturing, and 
other industries are the applied fields where AR competitively thrives and expands the scope of the technology itself. 
Particularly, according to their studies (Henderson & Feiner, 2009), which concentrate on the military sector, with 
the assistance of AR technology, military mechanical staff can conduct their routine maintenance tasks in a 
bulletproof vehicle more safely and conveniently. To do this, there are several required devices and apparatuses such 
as a tracked head- worn display to augment a mechanic’s natural view with text, labels, arrows, and animated 
sequences designed to facilitate task comprehension, location, and execution. 




a b 



Figure 8. a) A mechanic wearing a tracked head-worn display performs a maintenance task inside an LAV- 
25A1 armored personnel carrier, b) The AR condition in the study: A view through the head-worn display 
captured in a similar domain depicts information provided using augmented reality to assist the mechanic 
(Henderson & Feiner, 2009). 



Conclusion 

The future of Augmented Reality as a visualization technology looks bright, as shown by the interest 
generated in business and industrial circles as well as discussed in popular periodicals and research papers in the 
learning and training fields. Many questions still linger in terms of efficiency and when compared to traditional 
methods, particularly given the investments needed in research and design. However, there is much optimism of AR 
in learning and training for the future. New technologies and information communications are not only powerful and 
compact enough to deliver AR experiences via personal computers and mobile devices but also well developed and 
sophisticated to combine real world with augmented information in interactively seamless ways. 
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The future of learning and training with AR 

Several cutting-edge AR applications to date have been mostly developed for location-based information, 
social network services, and entertainment. New AR tools for other purposes such as learning and training, however, 
will continue to be developed as the technology becomes more highly evolved and advanced than ever before. A 
considerable number of professionals and researchers from the field of learning and training science predict that 
simple AR applications in education will be realized within a few years. 

Interactive education 

It is highly likely that AR can make educational environments more productive, pleasurable, and interactive 
than ever before. AR not only has the power to engage a learner in a variety of interactive ways that have never been 
possible before but also can provide each individual with one’s unique discovery path with rich content from 
computer-generated three dimensional environments and models. 

Simplicity 

As shown in a great deal of previous research and professionals’ opinions, AR could probably be focused on 
simplicity and ease of providing learning and training experiences, so that students and trainees can accept 
knowledge and skills with 3D simulations generated by computers and other electronic devices. In addition to that, 
related industries and technologies, such as computer and mobile industries, information and communication 
technologies, and Internet network infrastructures, including both wired and wireless services, possibly enable AR in 
learning and training to be much more straightforward and succinct to approach and utilize than ever before. 



Contextual information 

In the view of many professionals and experts in the field of educational AR, it is possible that learning and 
training-oriented AR can improve the extent and quality of information in both school and business settings by 
making learning and training environments more educational, productive, and contextual. In this perspective, there 
seem to be many contextual elements possibly embedded in educational AR applications in order to enhance the 
quality of learning and training by producing and delivering rich, constructive, and gainful content. For instance, 

Geo tag information for historical and cultural heritages could be connected as well as annotation regarding complex 
physical objects and artifacts could readily be added to AR tools in both business and school venues. 

Efficiency and effectiveness 

There is the potential that AR can promote the efficiency of learning and training in academic and 
corporate surroundings by providing information at the right time and right place and offering rich content with 
computer-generated 3D imagery. AR may appeal to constructivist notions of education where students take control 
of their own learning and could provide opportunities for more authentic learning and training styles. Besides, there 
are no real consequences if mistakes are made during skills training in terms of dangerous and hazardous work 
environments. As the results of several studies have shown, AR systems can provide motivating, entertaining, and 
engaging environments conducive for learning. In addition, AR applications in educational settings are attractive, 
stimulating and exciting for students and provide cost-effective support for the users. 
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Redesigning a Teacher Education Program by Critically Examining the 

Foundation of the Enterprise 

Kenneth J. Luterbach 

This paper considers a macro-instructional design problem, particularly the redesign of the foundation of a 
teacher preparation program. First, this paper discusses the foundation of teacher education programs in general. 
Then attention shifts to a particular case. This paper describes the initial redesign of the teacher preparation program 
and then discusses initial implementation and evaluation efforts. 

What is the Foundation of Teacher Education Programs? 

Teacher education programs are founded on conceptions of teacher capacity, which is the combination of 
knowledge, skills, and dispositions that teachers need to possess in order to help K-12 students learn (McDiarmid & 
Clevenger-Bright, 2008). Since conceptions of teacher capacity change over time, teacher educators at public 
institutions have a duty to ensure that their programs are commensurate with the times and responsive to the citizenry. 

A key challenge, as history reveals, is that citizens hold vastly different views about the purpose of schooling. 

Classicists, like Robert Flutchins (1936), favor the acquisition of “universal truths” through rigorous study 
of the classical curriculum (rhetoric, logic, and mathematics). Progressives oppose that view and, consequently, are 
perceived by classicists as a threat to intellectual rigor. Progressives like George Counts (1932), favor a social 
reconstruction agenda, which rejects capitalism and the power held by those in what they view as the failed and 
elitist class. Flumanists advocate for a grade school curriculum that favors a quest for self-actualization, self- 
fulfillment, and realizing one’s potential (Rogers, 1951; Maslow, 1971). Taylorists and others interested in labor 
productivity and scientific management for economic efficiency are primarily interested in preparing youth for the 
job market (National Commission on Excellence in Education, 1983). 

Tensions abound between those who promote humanistic/progressive positions and supporters of 
utilitarian/classicist perspectives. In this light, resolving the dispositions battle seems nearly intractable, but teacher 
educators and others may still agree that consistent application of a small and varied set of instructional strategies or 
methods may fulfill the obligation of teachers to help K-12 students learn. 

Redesigning the Foundation of a Particular Teacher Preparation Program 

Teachers are instructors; they help people learn. In K-12 public schools, teachers direct their efforts toward 
helping students attain particular instructional objectives, sometimes called goals or standards, identified by state 
departments of public instruction. The North Carolina Standard Course of Study 

(http://www.ncpublicschools.org/curriculum/) exemplifies such a set of goals. While the end goals are clear, the 
instructional means for achieving the goals are much debated, which leads to the key question: Flow should teachers 
instinct students? 

A team of teacher educators has been pursuing that question. The team formed in 2009 after acquiring a 
Teacher Quality Partnership (TQP) grant through the Office of Innovation and Improvement at the U.S. Department 
of Education. Grant resources permit three teacher educators at East Carolina University (ECU) to spend half their 
time on curricular reform issues. Each teacher educator has a unique specialty, namely elementary education, middle 
grades education, and special education. Consistent with the approved grant application, the team’s goal is not to 
modify every course in the teacher education program, but to ensure, as a condition of graduation, that all teacher 
education students know a particular set of instructional strategies; can create lesson plans that use them 
appropriately; and can use the instructional strategies effectively when teaching. 

Seeking a Core Set of Instructional Strategies 

In the first year of the grant, the design team defined instructional strategy as “an educational practice that results in 
learner achievement” and then resolved to identify a core set of research-supported instructional strategies that all 
teacher education students in the program should learn. Throughout the search for a set of instructional strategies, 
the design team recognized that the effectiveness of any instructional strategy implemented by a teacher in a K-12 
classroom depends in part on classroom management strategies and practices. For example, when teachers wish for 
students to attain instructional objectives collaboratively, grouping strategies, and effective implementation of those 



411 




strategies, become important. Also important to effective teaching is a disposition favoring reflective practice. 
Teachers do well to reflect on their efforts in light of student evaluation data, which reveal the extent to which 
students learned. Given the results of numerous research studies, the design team also regarded teacher questioning 
strategies, graphical representations, and concept learning as important to effective teaching. In this light, the design 
team decided to help teacher education students learn about grouping students; questioning students; methods for 
concept learning; graphical representations; and assessment. 

Implementation and Formative Evaluation 

Having settled on particular categories of teacher knowledge, the design team created instructional materials to help 
students learn the following instructional strategies: think-pair-share; jigsaw; teams, games, and tournaments; high 
level questioning; advance organizers; compare and contrast; examples and non-examples; formative evaluation; 
and summative evaluation. Students in their first course in education, called the Early Experience Course, received 
a handout that provides an overview of each strategy. The overview includes a definition, which identifies the key 
features of the strategy; commentary on the benefits of the strategy; and an example that describes how a teacher 
could use the strategy. 

After the Early Experience Course, teacher education students in Elementary Education, Middle Grades 
Education, and Special Education encounter the strategies again. During the second encounter, students are required to 
complete a custom-made online tutorial pertaining to the strategies. The tutorial requires students to read about the 
strategies and to respond to approximately four quiz questions per strategy. In addition, students complete an 
instructional strategies test before logging into the online tutorial for the first time and then take the same test after 
completing the tutorial. Although the tutorial is entirely online, the pretest and posttest are administered face-to-face. 

At ECU, a formative evaluator assesses TQP project activities and provides recommendations intended to 
improve project work. The formative evaluator (and author of this article) is employed at ECU as a faculty member 
in the Instructional Technology program area. Course reform, to ensure that students graduate with knowledge of 
particular instructional strategies and skill to use them effectively, is a fundamental feature of the TQP project at 
ECU. Accordingly, the formative evaluator has reconsidered the original list of instructional strategies on multiple 
occasions. 

After the pilot testing concluded, the instructional designer interviewed the instructors who taught the 
original set of instructional strategies. Analyses of those data and comments by the team members lead the 
instructional designer to recommend modifications to the student handout containing the instructional strategies. 
Currently, the instructional designer is promoting the adoption of Instructional Strategies 2.0, which embeds the 
instructional strategies in a teaching process that includes planning, implementation, and evaluation. 

When planning a lesson, teachers select particular instructional strategies in order to help learners achieve 
the instructional goal. During implementation, teachers follow their plans, at least initially, in order to engage 
learners. Teachers try to engage learners using multiple methods. An instructor may proceed through a series of 
steps somewhat similar to Gagne’s 9 Events of Instruction (Gagne, 1985) or Rosenshine’s Six Functions of 
Instruction (Rosenshine, 1986), which are consistent with behaviorism. Following such a sequence a teacher may 
speak to all students in the class in order to gain their attention; help them recall prior knowledge; present new 
content; require that students respond to instructional stimuli; and provide feedback to students. While engaging in 
this form of instruction over multiple lessons, a teacher would use a variety of instructional methods, which may 
include, for instance, presentation of an advance organizer; use of graphical representations; comparing and 
contrasting; and use of examples and non-examples. 

In contrast, a teacher may employ an inductive or Constructivist instructional strategy (Armstrong, 1991; 
Collins, Brown, & Newman, 1989). Guided by constructivism, a teacher may employ an inquiry approach, which 
may require a student to solve or address a problem, perhaps by acquiring and analyzing diverse perspectives; 
selecting one perceived as best for addressing the problem; and defending the selection. Another possibility is for a 
teacher to direct students to a tutorial, a Webquest perhaps; or have the students work on a project requiring analysis 
and synthesis (e.g., creating a multimedia presentation intended to persuade viewers to make a particular choice). 
While offering this form of instruction over multiple lessons, a teacher would use a variety of instructional methods, 
which may include, for instance, Problem-Based Learning; Cognitive Apprenticeship; Authentic Instruction; and 
High-level questions. 

After implementation, instructors evaluate student learning and may refine their teaching practices by 
engaging in formative evaluation and reflection. At the end of a unit, semester, or year, teachers or administrators 
may also conduct summative evaluations in attempts to discern overall student achievement and impact of their 
instruction. 



412 




References 



Armstrong, E. (1991). A hybrid model of problem-based learning. In D. Boud & G. Feletti (Eds.). The challenge of 
problem-based learning, (pp. 137-149). London, UK: Kogan Page. 

Counts, G. (1932). Dare the school build a new social order? New York, NY : The John Day Company. 

Collins, A., Brown, J. S. & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing 
and mathematics. In L.B. Resnick (Ed.), Knowing, learning and instruction: Essays in honor of Robert 
Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum. 

Gagne, R. M. (1985). The conditions of learning and theory’ of instruction (4th ed.). New York, NY : Holt, Rinehart, 
& Winston. 

Hutchins, R. M. (1936). The higher learning in America. New Haven, CT: Yale University Press. 

Maslow, A. H. (1971). The farther reaches of human nature. New York, NY: Viking Press. 

McDiarmid, G. W. & Clevenger-Bright, M. (2008). Rethinking teacher capacity. In M. Cochran-Smith, S. Feiman- 
Nemser, D. J. McIntyre, & K. E. Demers (eds.), Handbook of research on teacher education: Enduring 
questions in changing contexts. (3rd ed.). (pp. 134-156). New York, NY: Routledge. 

National Commission on Excellence in Education (1983). A nation at risk: The imperative for educational reform. 
Washington, DC: U.S. Department of Education. 

Rogers, C. (1951). Client-centered therapy: Its current practice, implications and theory. London, UK: Constable. 

Rosenshine, B. V. (1986). Synthesis of research on explicit teaching. Educational Leadership, 43 (7), 60-69. 



413 




Embodiment and embodied learning in online learning environments 



Lina Metlevskiene 
Kasnas State University 
linam@ksu.edu 



College of Education 
Department of Curriculum and Instruction 
261 Bluemont Hall| 

1100 Mid-Campus Drive 
Manhattan, KS 66506-5301 



Descriptors: Embodiment, online learning, narrative 

By signing below, I assign copyright of this manuscript to AECT and the Research and Theory Division of AECT. 



Abstract 

Recent findings in cognitive science and phenomenology indicate that embodiment and engagement in 
embodied practices facilitate learning. Are embodiment and the embodied interaction possible in online learning 
environments, where bodies are invisible and the direct perceptual grasp of the communication partner is 
impossible? This paper analyzes how the lived body can be articulated and experienced in a text-based medium. It 
presents narrative and the use of the language of literature as promising techniques that can engage learners in the 
embodied experience of others and foster embodied learning online. 

Introduction 

Many researchers, philosophers and theorists (DalTAlba & Barnacle, 2007; DalTAlba &Barnacle, 2005; 
Barnacle, 2009) in the field of education and curriculum criticize the conventional, rationalistic epistemology that 
currently dominates higher education. DalTAlba & Barnacle (2007) indicate that such epistemology is 
fundamentally flawed, because it puts too much emphasis on the development of intellect, knowledge, and skills 
“decontextualized from the practice to which they relate” (DalTAlba & Barnacle, 2005, p.680). However, the 
empirical research demonstrates that students have difficulty integrating decontextualized knowledge and skills into 
real practice (DalTAlba & Barnacle, 2005). DalTAlba &Barnacle (2005) point out, that this focus on a narrow 
concept of the intellect is the tradition of Western epistemology that comes from the Descartes mind-body division. 
According to Barnacle (2009), it renders body and embodiment epistemologically unimportant to learning and 
knowing. Nevertheless, recent findings in cognitive science and phenomenological inquiries reveal a less 
hierarchical view of the body and mind’s relationships. In addition, these findings point out that our embodiment 
and embodied practices facilitate our learning and knowing (Gallagher, 2008; Gallagher & Hutter, 2008; Dreyfus, as 
cited in Barnacle, 2009). Gallagher (2008) indicates that infants through such embodied practices as attuning to and 
the imitation of the caregiver’s voice, gaze, and bodily movements develop “primary subjectivity” - the ability to 
have an instant, pre-conceptual, pre-reflective understanding of others (p.539). Phenomenologist Hubert Dreyfus (as 
cited in Barnacle, 2009) in his account demonstrate that the development of informal forms of knowing through 
embodied practices forms the basis for formal, propositional, and conceptual knowledge to develop. If the body and 
embodiment have such a critical role in learning, what about text-based computer mediated communication (CMC) 
and online learning? Is embodiment and embodied learning possible in text-based CMC, where physical bodies are 
invisible and learners’ identities are articulated only via text? 
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Embodiment Turn 



There are two trends of embodiment that, in fact, complement each other. The first one is the embodiment 
of mind in body. This aspect of embodiment indicates that there is no clear division between mind and body, i.e. the 
body embodies the mind, and in the act of perception they both function together as one lived body. Merleau-Pointy 
(2008) was the philosopher who returned the significance to the body. Through his assertion that “it is the body that 
perceives” (p.57), he overcame Cartesian Mind-Body dualism. In fact, he heralded a new concept of the body - a 
phenomenal body that is comprised neither from body nor mind but is constitutive of both. According to him, the 
body is the ground for consciousness to develop. Thus, to be able to project itself into the environment, 
consciousness always needs the body: “Consciousness is being-towards-the-thing through the intermediary of the 
body” (Merleau-Ponty, 2008, p. 159). Another aspect of our embodiment indicates that our bodies are embedded in 
the world. In Merleau-Ponty’s account (1968), this aspect of embodiment is explained through the metaphor of 
“flesh.” According to him, our bodies are made of the same flesh as the world and this is the primary reason why it 
is possible for humans to have this “brute or wild perception of the world” (Burkit, 2003, p. 327). Also, our ability to 
perceive the world lies in the reversibility phenomenon of the flesh. This reversibility is enacted through two aspects 
of our embodiment in the world: one, as a subject, a sentient being who can touch, see, feel, and perceive others, and 
another, as an object, a sensible being that can be touched, seen, and perceived by others. As we engage in 
interaction with the world, there is constant reversibility between these two aspects of our embodiment. Thus, this 
“double belongingness” to the order of the object and subject and this constant reversibility between them, according 
Merleau-Ponty (1968) provides us with a direct, pre-reflective, pre-conceptional perception of the world, others, and 
ourselves. 



Embodiment Online 

Are embodiment and embodied knowing possible online? The literature of the 80s and 90s points out the 
inevitable division of mind and body online. The “body [online] vanishes in fluid simulation” (Tripathi, 2005) and 
consciousness can be “downloaded” (Bayne, 2004, p. 106). From a phenomenological point of view, the idea of 
disembodiment online is a misleading one. Merleau-Ponty’s (2005) account indicates that without the body we have 
no place from which to perceive the world. Therefore, the loss of embodiment online would mean the loss of our 
ability to recognize and make sense. Land (2004) indicates that the theme of disembodiment has arisen online 
because of the lack of clear distinction between the physical body and the concept of embodiment. Usually, in 
cyber-research literature, embodiment is associated with the physical body and visibility; whereas, from the 
phenomenological point of view, embodiment is about engagement with the phenomenal body that, according to 
Feenberg (2003) “has amazing plasticity” (p.107) and can be extended linguistically in textual CMC. Thus, the 
embodiment and the body’s existence online can be explained through the extension of our bodies via instalments or 
artifacts. According to Idhe (2002) and Merleau-Ponty (2008), our perception, embodied through the instruments 
and artifacts, extends our bodies into the world and allows them to experience new, previously unavailable 
meanings. Flowever online learning environments are predominantly textual. While extending some bodily 
perceptions as vision, they reduce, exclude, or distort others. As result, online learning environments do not allow 
“full bodily sensory awareness” to take place and, therefore, bring perceptually-reduced forms of embodiment 
(Bauer 2004, p.86). Flow does this perceptually-reduced form of embodiment impact our communication and 
learning online? 



Non-neutrality of Technology 

The perceptually-reduced embodiment in textual CMC brings in the issue of the non-neutrality of 
technology. Because online technology does not allow full bodily experience, it, according to Dall’Alba & Barnacle 
(2005), offers affordance for certain kinds of activities and presences, while closing possibilities for others. As a 
result, it promulgates “particular ways of knowing and acting” (p.737) and particular ways of being that for some 
learners can be an enabling and for others a disabling learning experience. In addition, not only do we use 
technologies, but technologies also use us (Dall’Alba & Barnacle, 2005). This is best described by Feenberg (2003), 
who indicates that technology extends to our bodies not only in an active, but also in a passive dimension, i.e. it 
signifies us and our bodies as having particular qualities. For example the blind man’s cane not only allows to the 
blind man to “see” but also it signifies his blindness for others. Online learning environments can also “signify” 
learners and teachers in new and unsuspected ways. The non-neutrality of textual CMC and its impact on teaching 
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and learning experience is nicely demonstrated in students and teachers accounts in Bayne’s (2004) study. Some 
learners indicated having difficulty learning without using their own bodies to communicate a message. Learners 
also voiced the need to see their communication partners’ emotional reactions. Not being able to see them, they had 
to re-construct emotional responses from the textual messages of their communication partners. This, according to 
Bayne (2004), made their communication into an “interpretative” act instead of an “intercorporeal” one. However, 
for other learners, the invisibility of bodies gave them the possibility to re-articulate their bodies differently and to 
explore and enact their alternative identities. As one student pointed out, the communication via textual CMC 
reduced the possibility of physical judgment. Online, “looking” was subordinated to “writing”. Thus, 
communicators were judged not by physical appearance, but by their writings. As a result, good-looking girls, 
instead of playing the role of attractive females, could explore their alternative identities; whereas shy students, who 
felt insecure speaking up in face-to-face class, in textual CMC were able to create new identities capable of speaking 
out (Bayne, 2004). 

These findings indicate that the textual medium can be liberating for some and limiting for others. 
Therefore, the effects of invisible online embodiment are not a matter of right or wrong (Enriquez, 2009), but for 
whom and in which context (Dall’Alba & Barnacle (2005). Hence, rather than treating technologies as neutral tools, 
Dali’ Alba & Barnacle (2005) call for the awareness of their differential impact. 

Problems with Online Learning Environments 

Gallagher (2008) argues that to understand other people in face-to-face interactions we always employ 
direct perception that allows us to have an instant pre-reflective grasp of other people’s intentions and emotions 
from their actions, behaviors, and expressive movements. According him, there is much available in the person’s 
movements, gestures, facial expressions. However, in textual CMC, where bodies are screened, learners are 
deprived of this direct perceptual grasp. In line with Gallagher’s (2008) thought, Quaeghebeur and Reynaert (2010) 
posit that intersubjective understanding takes place more easily in face-to-face then in textual communication. 
According them, in face-to-face interaction we use language in an embodied way - we engage in an “embodied 
linguistic spectacle” that is simultaneously constituted of “thought”, “verbal utterance”, and “bodily expressivity” 
(p.25). The verbal mode of communication there is not “the central conveyer of meaning” (p.25). It is greatly 
supported by bodily expressivity - gestures, facial expressions, gaze, intonation, etc. As result, in face-to-face 
interaction meaning is “overdetermined” (p.31), whereas in pure verbal (text-based) communication, there is 
possibility of a gap between the semantic and pragmatic meaning. This gap that, according to Quaeghebeur 
&Reynaert, can only be closed by “bodily expressivity” leaves textual communication prone to misinterpretations 
and misunderstandings. 

How do the limitations of the written language and the lack of direct perceptual grasp affect our 
communication and learning online? Becker (2008) points out that the inability of the textual medium to express 
deep emotions and feelings brings faulty feelings of successful communication and the illusion of consensus that 
frequently occurs online, where simple and rapid textual interchanges are used. Erdinast-Vulcan (2007) is 
concerned with the fact that textual communication is not ethical enough. According him, ethical communication 
does not allow the “assimilation of the other into the Same,” but “leave[s] the alterity of the other untouched, 
irreducible” (p. 398). Referring to Merleau-Ponty’s philosophy, he indicates that communicators can experience the 
alterity of the other only in concrete, temporal, and spatial situations, i.e. in face-to-face encounter where direct 
perception can take place. Communication via textual CMC is not an “incorporeal”, but “interpretative” account 
(Bayne, 2004, p. 110), therefore, there is a potential danger to undermine the ethics of communication, i.e to miss 
the alterity of others by reducing it to sameness. 

Becker (2008) talks about a similar problem, but he conceptualizes it as the lack of communication 
responsibility. According to him, in textual CMC communication, there can be the lack of communication 
responsibility because of the absence of physical touch and eye contact. By responsibility he means recognizing and 
accepting the strangeness and particularity of the other “without trying to integrate the other in their personal 
horizon of images and expectations” (p. 174) and accepting associated boundaries to their own understanding of the 
other. As Becker (2008) argues, touch and eye contact allow us to recognize the “unavailable otherness” of our 
communication partner because “glances demand recognition and respect” as well as “place limits on occupative 
desire” (p. 172). However, in text-based CMC, individuals are deprived of this direct perceptual grasp (Gallgher, 
2008). As a result, in online settings, they are inclined to understand the other only from their point of view. This 
prevents seeing the “unavailable otherness” of the other and, literally, reduces the other to the object ( Becker, 2008, 
P- 174). 
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These statements are nicely exemplified by Henriksson’s (2003) account, where she analyzed her 
experience as a student in an online classroom. She found that in the online learning environment, she could easily 
disregard the need to see the teacher’s body until she met her teacher face-to-face. Then she realized that, in fact, she 
did not know him, that “out of a person-as-text” she had had created “a person-as-idea” and took it to be a real 
person. Analyzing her experience, she recognized that online she was not “thirsty” for the teacher’s body, because 
she was “content with the body of knowledge presented by” him (p.20). It means that she treated him as an object, 
as a “means to an end,” i.e. engaged in “I-It relations” (Buber, as cited in Henrikson, 2003, p. 20). Henriksson 
(2003) warns that, in online learning environments, our created illusory images of others might close off the 
possibility to interact with “the whole person” and feel thirsty for other I-Thou (mutual) relations. Thus, she points 
out the need to teach online learners to doubt the images of communication partners they have created online (p.21). 

Lived Body Online 

If the concrete embodied experience of others is so vital for genuine intersubjectivity to take place, is there 
any possibility in textual CMC to compensate for the lack of physical touch and look as well as of other bodily 
experiences? Can lived body experiences be sustained online? Feenberg (2003) indicates that because of its 
complexity, language contains huge compensatory efforts that allow compensating for the lack of full bodily co- 
presence: “Our language shows us as neat or sloppy, formal or informal; we reveal our mood by our linguistic 
gestures as happy or sad, confident or timid” (p.107). 

Some research studies indicate that, in textual CMC, online learners extensively use linguistic bodily 
expressions. This is in line with Albano’s (1998) remark about inevitable “corporeal metaphoricity in all utterance” 
(p.387) due to the tremendous role of body in communication and understanding. Lander (2005), analyzing learners’ 
textual interchanges, found plenty of expressions of feeling, eating, speaking, and consuming body. For instance, 
students “go over to chat room”, “they grasp ideas” (p. 163), they share their momentary experiences of drinking 
coffee and taking care of babies while typing, they send each other cyber hugs and kisses, etc. 

Other research studies indicate that learners experience CMC bodily. As in the case of the online student who 
“blushed in front of the computer” when she realized that her lurking was visible to everyone because of the design 
features of Web Board. She “felt like [she]. . .had sneaked around in a room full of people trying to hide behind 
furniture, thinking [she]. . .was invisible, when, in fact, everybody saw [her]” (Flenriksson, 2003, p. 16). Robinson 
(2000) in her phenomenological study found that in the salient textual medium students were able to “hear,” “see” and 
“touch” each other. In fact, they did not read messages but listened to them. Through creative spelling and the tone of 
the writer, they could hear the voice of the text’s writer. The possibility to hear voices promoted a better connection 
with others and clarified “the meaning behind the writing.” In fact, it was through the informal tone that they could 
“hear” the voice of students, whereas, a formal tone did not contain the voice of writer and, therefore, let them “bypass” 
each other (p.252-255). Online students also scrutinized textual messages searching “for images or mental pictures” 
(p.255). Additionally, students indicated that they were able to extend the physical feeling of touch to a virtual one. 
They touched each other through the stories shared. This touching was critical, because it allowed them to reach 
something deeper in each other and the course content Thus, Robinson’s study indicates that the lived body can be 
articulated in text-based CMC. And the more learners are able to create lived bodies in a text-based medium, the easier 
it is for them to connect with each other and the course content. Not all writings, however, lend themselves to this 
purpose. As Robinson points out, only sensuous personal narratives that contain a rich, vibrant, individual tone and a 
personal, touching storyline allow learners to bring lived body experiences online. 

Lander (2005) agrees with Robinson’s findings that lived body experience can and needs to be sustained 
online. According to him online technology calls for new “focal practices” - for re-embodiment online through the 
deliberate work of “re-membering of the body”(p. 159). Referring to Weneger’s (2008) seminal work “Communities of 
Practice,” he posits that re-membering of the body online requires “the work of engagement and the work of 
imagination” (p. 166). Learners have to engage in the work of articulating their lived bodies online as well as engage in 
the active work of imagination - of “reading”, “seeing”, and “touching” lived bodies of others in a textual medium. 
According to Lander (2005), the act of re-membering of bodies online can be achieved through several processes. One 
of them is the act of virtual hospitality such as “welcoming strangers” into the community, “modeling greetings,” and 
“drawing in a partaker after some time [has] elapsed without a written response” (p. 169). He also points out that 
“bracketing the feelings dimensions” (p.169), particularly emotional information will allow keeping the body present 
online. The third process that sustains lived body experiences, according to Lander, is a narrative technique. Along the 
same lines as Robinson, he also highlights the role of “the bodily depended, imaginal practice of telling a personal 
story” (p.169). 
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Why Narrative is the Best 



Why is narrative technique suitable for articulating lived body experiences in a text-based medium? 
According to Gallagher and Hutto (2008), narrative gives a more complex and nuanced understanding of others. It 
allows us to understand the reasons why people act, think, and feel in particular way as they do. Some scholars in 
cognitive science point out that simulation of minds - the access to “landscape of consciousness” (p. 14) - is the way 
we understand others’ reasons. However, according to Gallagher and Hutto (2008), what we seek to understand is 
“much richer” (p. 13) than mental states. It is the reasons for the actions of a whole situated person. Therefore, those 
reasons for action need to be contextualized in terms of cultural norms and a particular person’s history, values, 
attitudes, etc. For this purpose, narrative is the best technique because the story line gives us access to a 
person’s “landscape of action” that entails person’s embodied actions in “rich worldly context” (p.14). In addition, 
according Gallagher and Hutto (2008), listening to the story is an embodied experience. It “presupposes a wide 
range of emotive and interactive abilities” (p. 12) that we have developed in infancy as embodied practice. Thus, to 
have a complex and nuanced understanding of others, learners need to develop narrative competence - the ability to 
express their lived body experiences in a narrative as well as the ability to decipher the narratives of others. 

Language 

The accounts of Feenberg (2003), Robinson (2000), Lander (2005), and Gallagher & Hutto (2008) point 
out that language might be capable of bringing that virtual look and touch through which lived body experiences and 
the uniqueness of the person can be communicated in textual CMC. However, not all kinds of language might be 
able to serve this function. According to Merleau-Pointy’s account, to serve such a role, language needs to be as 
expressive as our bodies are. 

Merleau-Pointy (Baldwin, 2007) distinguished between two kinds of language use: “spoken speech” and 
“speaking speech.” The “spoken speech” is ordinary, everyday language as well as formal, scientific language. It 
uses established modes of expressions that do “not attract our attention” and, therefore, “permit us to pass 
effortlessly” to communicated truths (Baldwin, 2007, p.92). The “speaking speech” is the use of the language in a 
creative and novel way, when new expressions are introduced. It is the language of the child, the lover, the poet, or 
the philosopher - in general the language of literature. As Merleau-Pointy indicates, the power of the speaking 
speech is not only in what is said, but also in how it is said. Merleau-Pointy sees a parallel between the expressivity 
of “speaking” speech and bodily expressivity. According to him, novel expressions are linguistic gestures that reveal 
“the whole truth about a man” in the same way as bodily gestures do (Merleau-Pointy, 1969/1974, as cited in 
Erdinast-Vulcan, 2007, p.89). Also, novel and unique expressions don’t let us pass effortlessly to the truth 
communicated. It brings a “coherent deformation” (Merleau-Ponty, as cited in Erdinast-Vulcan, 2007, p. 403) or 
“defamiliarization” phenomenon (Shklovsky, as cited in Erdinast-Vulcan, 2007, p. 403) that breaks down our 
“habitual modes of perception.” And this very impediment of our perception, inhibits “the smooth assimilation and 
reduction of otherness. . .it demands that we recognize an otherness” in others and ourselves (Erdinast-Vulcan, 2007, 
p.403-404). Therefore, the use of literary language or, more specifically, the use of novel and unique expressions 
might allow learners to bring their lived bodies online. It might allow for virtual look and touch to take place in a 
text-based online medium, so a learner’s uniqueness, alterity, and unavailable otherness can be articulated and 
experienced by others. 



Conclusion 

Accounts in philosophical and cognitive science indicate the crucial role of the body, embodiment and 
embodied practices for embodied learning and knowing to take place. This paper analyzed if embodiment and 
embodied learning is possible in textual CMC. As different authors indicate, learners do not lose their bodies online. 
Through perception their bodies are extended there. Thus, learners remain embodied in textual CMC. In addition, 
they use many linguistic expressions of the body and can even experience their communication online bodily. 

According to Dall’Alba & Barnacle (2007) for embodied learning to take place, learners need to be 
engaged in “intercorporeal subjectivity”, i.e. to meet bodies. However, in textual CMC bodies are invisible. As 
result, the direct perceptual grasp that would bring concrete and embodied experience (Gallgagher, 2008) of 
communication partners is unavailable. In textual CMC learners experience each other only through writings. As 
result, their communication is an interpretive act, instead of being an intercorporeal one (Bayne, 2004). These facts 
point out that the invisible embodiment online might not be enough for embodied learning and knowing to take 
place in text-based CMC. Nevertheless, other authors (Merleau-Ponty, 2005; Robertson, 2000; Lander, 2005; 
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Erdinast-Vulcan, 2007) point out that language has the potential to articulate lived body experiences online. The use 
of literary language, more specifically, the use of creative and novel expressions, turns language into linguistic 
gestures, which in the same way as bodily gestures, can engage learners in the concrete and embodied experience of 
the other. The coherent deformation phenomenon might bring this virtual look and touch through which true 
uniqueness, alterity, and the unavailable difference of the other can be articulated and experienced. Thus, the 
language of literature, in textual CMC, might engage learners in a virtual intercorporeal subjectivity. The use of 
personal and engaging narrative is another way to experience embodied interaction online. According to Gallagher 
and Hutto (2008), narratives give access to embodied actions and the embodied experiences of others. The reading 
of a story is also an embodied experience. It necessitates readers emotional and sensory-motor responses. Burbules 
(2004) points out that reading or listening to the story can be a very immersive activity, and, therefore, is 
experienced bodily. Thus, through the sharing stories learners can engage with the body in textual CMC. 

However, online learners might have different perceptions about the purpose of online discussions. They 
might engage in rapid and simple internet exchanges or in formal writings - the monologues - through which lived 
body experiences cannot be communicated. Also, they might not be cognizant of the necessity of “reading” the 
lived bodies of others online. Therefore, online learners need to be informed, scaffolded, and modeled to engage in 
the process of re-membering the body online: (1) to bracket their feelings and emotions, (2) to share personal 
stories, (3) to use novel, unique, and artistic expressions, as well as (4) to “read”, “see”, and “touch” the lived 
bodies of others. On the other hand, not all learners might be skilled in articulating their lived bodies as well as 
“reading” lived bodies of others online. As a result, there is still the possibility that online learners will undermine 
communication ethics and communication responsibility to see the alterity and unavailable otherness of the other. 
This problem, according to Becker (2008), might be offset by learning appropriate “media competence” - teaching 
learners to be aware of the “projective-illusory dimension of communication,” (p. 178) instructing them to doubt 
their images created, as well as helping them to recognize the boundaries of their own understanding of the other. 
Blended learning environments, where in face-to-face meetings learners can obtain concrete and embodied 
experience of each other, can also offset this problem. 

However, embodied learning and knowing is not only about engagement with the body. According to 
DalTAlba & Barnacle (2007), it is more about pedagogy: about engaging learners in a rich worldly context, into 
practice with questions, where they can develop “responsive spontaneity” and “appropriate ways of being in the 
world” (p.685). From this point of view, to promote embodied learning online we do not need “to emulate ordinary 
multi-sensory experience” online, but to promote it via action: through the engagement into inquiry with questions 
as well as into collaborative activities with learners, experts and stakeholders (DalTAlba & Barnacle, 2005, p.740). 
According to researchers and practitioners in the field of online learning (Hung, Lim, Chen & Koh, 2008; Norton & 
Hathaway, 2008), the online learning environment lends itself well to implementing collaborative learning between 
remote learners and to giving access to resources and distant communities, experts and stakeholders. Thus, it is up to 
the teacher how they will design the learning experience. Embodied learning and knowing can successfully take 
place in online learning environments; however, online technologies can also be “successfully” used to to“extend 
the decontextualized nature of conventional programs into the realm of online environments” (DalTAlba & 

Barnacle, 2005, p.728). 
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Abstract 

This paper will focus on practical examples of how creative use of technology through free new 
media tools and thoughtful course design can help engage Asian students and stimulate class 
discussion among students who are accustomed to professor-led instruction. This research data 
comes from student feedback in different courses at Hannam University in Daejeon, South Korea. 
The strategies shared in this paper have been successful in transforming a class of bright, but quiet 
listeners into a community of engaged students in a participative learning environment. Although 
most international professors realize that the silence in classrooms is due largely to respect, 
culture, habits and expectations based on previous years of schooling, it can create a tension in the 
classroom. This tension can lead to frustration by both students and professors. The aim of this 
paper is to share details of how this tension can be softened. 

Keywords: constructivist, participatory education 



Introduction 

A Korean proverb suggests, “Even if a bridge is made of stone, make sure it is safe.” Perhaps this applies in 
the classroom as students often need to know that they are in a safe environment before speaking up. University 
administration may be wary of trying new pedagogy when the current system is not broken. The Organization for 
Economic Co-operation and Development (OECD) ranked Korea as a leading country in math, science and reading 
(OECD, 2010). In these settings the professor does nearly all of the talking during the class session. Outside of the 
classroom, Korean students are also traditionally far more reserved than North American students; verbally, their 
feelings are neither freely nor openly expressed (Samovar, 2001 p. 146). Even when Korean students are prepared 
for class, this silence in the classroom can pose a problem for international professors who are generally more 
exposed to more participatory learning. Students may be following the customs of generations before them who 
have received such advice as the Chinese Confucian proverb which advises to “think three times before you act.” It 
is important for professors to learn and respect the cultures that contribute to silence within the classroom. Only 
then, can they design successful plans for improvement. 

Regarding change in the classroom, a school principle for 29 years at one of Korea’s most prestigious 
foreign high schools wrote, “As a result of the high value placed on doing things the ‘Korean Way’, diversity and 
differences are often considered to be negative” (Borden, 2003, p. 29). Many international professors will find 
familiarity in that comment. Yet Kim and Fisher (1999) found that Korean students respond favorably to 
constructivist-style learning environments in the subject of science. They wrote, “this integration of approaches 
presents a difficult challenge, as the roles the instructor must play - an all-knowing instructor vs. a participant in a 
democratic learning process - can be in conflict” (Nilsen & Purao, 2005, p. 6). 

In 1977, Zaltman & Duncan wrote “Strategies for Planned Change”. Although it is beyond the scope of 
this study to discuss these elements in detail, it is important to state that they listed culture as the first major barrier 
to change in education. They broke down the “culture” barrier into four sub-barriers: values and beliefs, cultural 
ethnocentrism, saving face, and incompatibility of the cultural trait with change (1977 pp. 68-72). Although I have 
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no desire to deem one culture’s method as better or worse than others, I would like to point out that the tension 
between cultures does exist in the classrooms. 

Of course both Korean society and the Korean education system have continually developed into a much 
more complex culture than it’s strong Confucian roots, there are many remnants of this Confucian belief system 
throughout the culture. These reminders can be seen within the current education system both in the classroom and 
administrative offices. Korean students traditionally learn in a lecture setting (Cumings, 2005). Although Korea has 
roots in both Confucianism and Buddhism, the impact of Confucianism became more profound during the late 14th 
century. “Its pragmatic life philosophy became the official ideology of the Yi dynasty for five hundred years, being 
institutionalized and propagated through the educational system and government examinations” (Park & Kloph, 
2004, p. 44). 

This descriptive study will share some of the strategies that have worked for one professor currently 
teaching in a Korean university. Over the course of two semesters, six courses have been transitioned from face-to- 
face instruction to blended learning environments. More important than the introduction of class websites, are the 
course design and teaching methods that have helped the transition from lecture-focused instruction to truly 
participative education that will prepare students for a globally-connected world where 21st century learning skills 
and digital citizenship will become increasingly necessary. Past struggles and potential future stumbling blocks will 
also be addressed. Many free digital tools and resources will be shared. Opinions will be shared from students 
during this ongoing transition period. These tools can be used to help teachers in any academic discipline help 
students achieve personal learning goals and prepare them for lifelong learning without compromising course- 
specific objectives. 

“We are currently preparing students for jobs that don’t yet exist, using technologies that haven’t been 
invented. ..In order to solve problems we don’t even know are problems yet.” When Karl Fisch typed these words for 
his YouTube video creation in 2007, he probably did not know that it would be read by a modest estimate of 6.5 
million people world-wide within two years (Hakkenberg, 2009). Mirellie Hakkenberg of Viralblog, a site that 
tracks viral ideas and follows social trends, reports that this video was viewed by over 130,000 times within 96 
hours after being posted on the growing YouTube website. Obviously, this statement and the supporting statistics in 
the video struck a collective nerve with citizens, politicians, business leaders and educators around the world. 

The students in this study are from six courses that I taught at Hannam University's Linton Global College 
(LGC) department. This department was developed in 2005 and currently has approximately 200 students. LGC, 
provides a study-abroad experience combining English immersion and international cultures with high global 
standards of academic excellence. All faculty members are international native-English speakers. Only English is 
used in the classrooms, extra-curricular activities and dorm rooms. Students are accepted based on either high 
English speaking ability or high English writing ability. All students must complete six weeks of a study-abroad 
experience before graduation from LGC. 

Most students have international career goals of either working in other countries or working with an 
international business within Korea. The students in this study are from the following six courses: Case Studies in 
Communications, Foundations of Organizational Communications, New Media Technology, Internet Media 
Production, Principles of Audio & Visual Communication, and Audio Video Production. Although there are 
international students in these classes (about five percent), this study will focus on the Korean students’ progress and 
perceptions. Permission has been granted by students to share their comments. All names are anonoymous names to 
protect the identity of my students. Most students create English names for classes at LGC, thus I am using English 
pseudo names for these Korean students. All pseudo names are consistent with the gender of the original student. 

This paper will focus on practical examples of how creative use of technology and free new media tools 
and thoughtful course design has helped me engage Asian students and stimulate class discussion among students 
who are accustomed to professor-led instruction. I am currently enjoying my fourth semester at LGC. I have 
integrated blended learning, constructivist and collectivist strategies into each of the nine courses that I have taught 
over the past 20 months. I have previously integrated technology successfully into debate and TOEFL courses in 
Korean middle school classrooms. 
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Pedagogy, Precess & Outcomes 



Pedagogy 

Due to the diverse cultural backgrounds and skill levels of the students in this descriptive study, paired with the 
school’s goal of helping students become more internationally-minded, it was my aim to help create a learning environment 
where student-citizens had a chance to share their experiences and learn from each others’ experiences. I strive to integrate 
constructivist and project-based learning (PBL) instructional design principles into my Korean university classroom in order 
to help students benefit from their existing strong relationships with others in the classroom. It is also my hope to use 
technology to bring the wisdom of experts around the world into the classroom. 

Tliis philosophy fits well with Duffy and Jonassen (1992), who suggested that students of varied skills and life experiences 
should work on tasks where collaboration and discussions lead to a negotiated understanding of field-specific truth. This is 
very appropriate for university students in the time of globalization. Students can leam from their classmates who have 
travelled and lived in different areas of the world. The philosophy also agrees with Jonassen’s later suggestion that 
constructivist conceptions of learning, “assume that knowledge is individually constructed and socially co-constmcted by 
learners based on their interpretations of experiences in the world. Since knowledge cannot be transmitted, instruction should 
consist of experiences that facilitate knowledge construction” (Jonassen, 1999, p. 217). 

At times, it feels as if the term ‘constructivist’ is still being constructed. I personally prefer the interpretation of 
Savery and Duffy (2001, p. 2), who have summarized it in three points: 

1. Understanding is in our interactions with the environment. 

2. Cognitive conflict or puzzlement is the stimulus for learning and determines the organization and nature 
of what is learned. 

3. Knowledge evolves through social negotiation and through the evaluation of the viability of individual 
understandings. 

In his book Cognitive Surplus: Creativity & Generosity in a Connected Age, Clay Shirky (2010) envisions a hopeful 
near- future where millions of educated, digitally connected volunteers trade in their surplus hours of free time to creatively 
and collectively solve large and small problems around the world instead of passively consuming media as we have in the 
past. “The good news about our current, remarkable age is that we can now treat free time as a general social asset that can be 
harnessed for large communally created projects, rather than as a set of individual minutes to be wiled away one person at a 
time” (Shirky, 2010, p. 10). Shirky calculates that the surplus of time among digitally-connected citizens is 100 million hours 
of human thought. This represents the equivalent of doubling the information currently available on Wikipedia 2,000 times 
each year (Shirky, 2010). These statistics support the need for educators to prepare students in all nations for 21st century 
learning skills such as critical thinking, problem solving, communication, collaboration and digital citizenship (Finegold & 
Notabartolo, 2011). Rheingold (2007) states that for young people to be prepared to create wise change in their future, they 
must be prepared with skills of using participatory media technology, adapting their ability to communicate online, and 
communicate visually. 

These recent trends coupled with access to new digital tools, have served to motivate teachers around the world to 
incorporate cooperative learning methods into the classroom. A key to developing a successful collaborative classroom 
environment is to foster positive interdependence. Chen (2006) states that creating a knowledge building culture requires the 
design of structured activities to form task-specific knowledge while creating conditions that foster collaborative building of 
knowledge. This can be difficult in a learning culture where students are accustomed to competing with their classmates. By 
creative course design, a grading system that rewards interdependence and clear communication, the students’ appetite for 
competition can be appeased. 1 will close this section with a call to action by Donavan Walling: 

hr today’s new media environment in which students blur the line between school-paced technologies and 
personal devices and engagement, educators find themselves standing on the railroad track facing a 
speeding high-tech train. They can stand pat and get run down. They can step aside and get passed by. Or, 
if they recognize the promise of tech-sawy teaching, they can swing aboard and join their students on a 
fascinating journey of discover (Walling, 2009, p. 22). 



424 



Process 



Wisdom With Wikis 

In the New Media Technology course, I wanted to teach the students the potential value of Wikis. I also 
wanted the students to learn the content of a reading about technology predictions for 2011. Students were given a 
reading full of technical jargon for which they had five days to read. The students came to class confused and hoping 
that there would not be a quiz. At that point, I divided the class into groups of two. Each group of two was assigned 
one specific prediction to focus on. The students were then introduced to PB Wiki, a free, private online wiki 
program. After instruction on how to use the program, students were asked to summarize their section of the reading 
and do additional research to add to what they learned in the reading. Students were given 48 hours to add their 
knowledge to a joint wiki page. They were told that if 1 was content with the progress of the joint wiki project, there 
would be an “open-wiki quiz”. If I was not satisfied with the progress of the project, there would be a “closed-wiki 
quiz”. At this point, I shared some strategies of how to successfully manage a wiki by including success and horror 
stories from my experiences. 

I prepared the lesson plan to allow the final 10 minutes of class to allow the students to have a group 
meeting about how they were going to tackle this problem. Most of the students had never contributed to a wiki 
before and none of the students had used this specific wiki program until 45 minutes before the assignment was 
given. At the beginning of the next class, I walked into the class and loaded the fresh wiki and presented it on the 
screen. After looking at the wiki approvingly, I handed out the quiz and told the students they were able to turn on 
their computer and use the wiki that the class had created. Thirteen of the fourteen students earned a 100% and one 
student received a 90%. As I collected the quiz papers, I announced that all students would be tested on the contents 
of the reading on the midterm test without the help of the wiki. While the students were still anxious about the 
undisclosed results of the quiz and the new challenge before them, I asked each student to stand up and share what 
they had learned in the past 48 hours about the content of the reading. Students confidently shared what they had 
learned, I clarified the key points. The individual students witnessed that they were surrounded by 13 classmates 
who possessed specialized knowledge that all students would need to understand for an important upcoming test. 
Each student also had a wealth of wisdom created by their classmates that could be used for further learning at their 
own pace at any time that fit their schedule. This student shared her appreciation for the ability to learn from her 
classmates, 

“When I was working on small class project explaining what author said on reading 2, I never 
expected that we were going to make an well organized wiki page. However, Julia made template 
of the page nicely and most people edit their font size and type coherently, everyone made the 
wiki page perfectly, not only the design of page, but also each of explanations was wonderful and 
it helped me to understand the reading enough (Student “Leanne”). 

The following student concurs with the value of a team leader when using wikis, 

“I was a bit confused about the place where I should publish my summary. However, as the leader 
sent to every student a message explaining the steps to follow in order to add his part to the whole 
project, it made things clear for me .Therefore, wiki is useful for any project team as it allows 
online Collaboration in addition, the professor asked each student to talk in few minutes about the 
major points of his part, which made me a little nervous as I'm not comfortable with public 
speaking. However, I was able to overcome my fear of public speaking and talk about my part 
confidently (Student “Charlize”). 



Adopt- An-Expert 

In an attempt to bring experts into the classroom while allowing student to personalize the contents of the 
textbook and lectures, I require all students to adopt an expert in the career field that the student hopes to pursue 
upon graduation. Major course objectives include introducing students to the strategies used for Internet marketing 
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and podcast production skills. Students in Internet Media Production class must “Adopt a Podcaster”. Students in 
New Media Technology must “Adopt A Blogger”. The students embark on a 13 week adventure of following the 
expert of their choice. The first two weeks focus on how to identify an online expert as well as an online ‘fraud’ who 
can not be trusted as a credible source of information. They must put this skill to immediate use because after the 
third week of the semester, students are not allowed to change their expert. They have adopted him or her. During 
the third week, the students are introduced to Google Reader and are shown how to add to the RSS feed to ensure 
that they have constant access to their experts’ most recent digital content. 

The adopted expert plays a large part in each student’s final grade. There is a final project and final paper 
based on what they learned from the expert. The project and paper combine for roughly 17% of the final grade. 

There are also pop quizzes where students are given a blank sheet of paper and asked to share what they learned 
from their adopted expert over the past two weeks. Because I have the URL of each adopted expert, the students 
know that they must be accurate. Questions on the midterm and final tests account for between 5% - 7% of those 
tests. Answers are truly individualized essay questions where they are asked about both the content of expert as well 
as strategies used to grow or engage their digital audience. Students are also allowed to blog about their adopted 
expert at any time during the semester if they choose to not blog about what was discussed in class. Before students’ 
final presentations, I announced that one of the final questions will be about the content of what one of the students 
presents about. This adds collaboration and accountability. Each student clearly gives their presentation and 
welcomes questions from the class. Each student is attentive when not presenting because they know that they may 
be tested on any of the information. This can be seen in the following feedback from the blogs written after the 
students hear their classmates’ final presentations, 

In Friday's class, we had top blogger presentation. Actually I was surprised because there were 
two classmates who chose same blogger with me. Anyway it was interesting. Tara presented about 
Chris Brogan, and I was impressed about Chris Brogan, because he takes very complex thing and 
speaks very clearly and simply (Student “Mary”). 

I can apply knowledge that I gain in the class to a real (world) (Student “Philip”). 

Outcomes 

Although my teaching style is significantly different than the Korean style of university instruction, 
response has been positive regarding satisfaction. This mirrors Kim and Fisher’s findings that Korean students can 
adjust and succeed in a constructive learning environment (1999). Only positive feedback is shared below because 
negative feedback equates to less than eight percent of collected feedback. The two biggest trends in negative 
feedback are 1) the lack of clarification and confidence in whether they have the correct answer, and 2) large 
amounts of homework. My approach to my course design is that there are multiple correct answers if the students 
can successfully communicate how they arrived at their opinion based on theory, facts and trends. This lack of ‘one 
right answer’ may be causing additional hours to the already heavy student workload. As mentioned earlier, gender- 
accurate pseudo names have been created to protect the identity of my students. The first quotation comes from a 
student who commented many times during the semester about the dilemma of trying to balance his desire to dig 
deep into some of our topics while keeping strong grades in other courses, 

When I started this class I had no certain expectations. I thought we will get to know something 
about the technical aspects of new technology. I do like the fact that our professor introduces 
many websites as possible to us. As I said some of them are really interesting. I wish I had more 
time for this class, because we are going to learn much more about new media technologies. 

Unfortunately I have to focus on other classes as well:-( (Student “David”) 

Other students found that they could actually save time and enjoy the learning experience more by sharing 
knowledge and learning from others, 

We celebrated our first class project. I was in a great mood because this first project gave me a 
confidence. Now, I’m totally ready for the next project (Student “Ryan”). 
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1 liked the real experience. The ideas for projects were appropriate to this class (Student “Russ”). 

(I) can more focus on this class because this class very active (Student “Lisa”). 

1 learned how to organize the project planning (Student “Taylor”). 

1 think it was great week actually, because of communication. I know our AVP class needs 
collaboration together, without teamwork, we can't get anything to make up. I'm really happy to 
get lots of information from each other. I feel sure we can get great result (Student “Todd”). 

When professor brought up a conversation topic, I could gather others' opinion. Especially, in 
Case study in communication class (Student “Kathy”). 

After a semester of trial, error, adapting and adjusting, I was very satisfied when the most hesitant 
student in my class posted this comment on her final blog of the semester. 

Everyone in class showed attention, interest and collaboration very well. It made good synergy 
effect to whole class I think (Student “Jenni”). 

Conclusion 

It is not my desire to make a judgement or value comparison between two unique continental approaches of 
education. This research data was collected in an effort to share details of how collaborative and self-directed 
learning can be implemented by professors who choose to pursue a participative learning environment in Korea. 

This descriptive study focused on students with a wide variety of life experiences both within Korea and 
internationally. The subject matter of the courses studied were all within the communications department at a global 
university. For these reasons, it is uncertain of the ability to scale the results across various subjects and other 
universities in Korea. However, the qualitative data included shows that many students have found value in utilizing 
both technology and constructivist course design within the Korean university system. Some potential problems 
include technology, schedules, and support from administration. Most of the strategies used do not need a computer 
for each student at the time of instruction. Yet a computer connected to the Internet will be needed at some point for 
many of these activities. It is crucial to have the support of administration at the university. Any professor will want 
support from administration before beginning wide-scale integration of constructivist pedagogies as teachers need 
training in cooperative learning techniques to allow for proper online participatory learning (Sharan, 1994, p. 57). 

It is also important for the professor to be aware of the advantages and potential disadvantages of participatory 
learning. Blended learning does require many additional hours from the teacher. In this case, between two to three 
hours are required for each class each week to manage the website and provide feedback for the 15-25 students. 
Although the sample size is limited, by creating truly participative classrooms focused on collaboration and 
personalized learning based on students’ passions and goals, students will be prepared and energized to continue 
their lifelong learning long after their degree is signed. 
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Abstract 

This MOON project addresses common misconceptions experienced by children and adults in astronomy 
education, especially understanding and recognizing the lunar phases from different vantage points around the earth. 
The project also fills a void in available instructional software targeted at this important area of astronomy 
education. More specifically, this project enables students to “observe” the Moon’s phases from the Northern and 
Southern hemispheres and to observe the relationship between lunar shape and orientation by manipulating the 
position of the Moon and observer. 



Introduction 

Research shows that the second most important topic in astronomy education, next to the Earth shape, is 
Moon phases. Research suggests that comprehending the Sun-Earth-Moon relationship is challenging for many 
people (Lelliott & Rollnick, 2010). Many studies have found several misconceptions about this topic. First, many 
erroneously believe that the lunar shape, as observed from earth, is determined by the size of the Earth’s shadow on 
the Moon (ecliptic explanation)(Barnett & Morran, 2002; Lelliott & Rollnick, 2010; Schoon, 1992). Second, many 
people believe that the Moon’s movement patterns are unpredictable (Trundle, Atwood, & Christopher, 2007). A 
third misconception concerns the rotational pattern of the Moon around the Earth. For instance, some believe that 
the Moon orbits the Earth daily, some yearly (Schoon, 1992). Moreover, these misconceptions about the Moon 
pertain to adults as well as children (Lelliott & Rollnick, 2010). 

Some studies have shown that state science standards frequently require an understanding of the Moon’s 
shape and location, but no states require students to understand the different perspectives of the Moon’s shape and 
orientation from the Northern and Southern hemispheres. (Sherrod & Wilhelm, 2009). 

The MOON project addressed these issues in astronomy education and compensates for deficits in 
instructional software that is widespread on the Internet and in the schools. The authors designed the educational 
software focusing on global lunar patterns. Its major advantage will be the ability to “observe” the Moon’s phases 
from both the Northern and the Southern hemispheres simultaneously. The MOON project also enables students to 
change the Moon’s position to find out similarities and differences between lunar shape and orientation. 

Global Lunar Patterns 

The global lunar patterns include the shape and orientation of the Moon in reference to various 
observational positions around the Earth. The following principles, relative to global lunar patterns, are 
demonstrated in the software project. 

Shape. Although the lunar shape varies from one day to another, it is similar at any particular day in both 
hemispheres. Moreover, the pace and direction of change is the same in the Northern and Southern hemispheres. 

Orientation. In the same hemisphere, the Moon’s left-right orientation is similar. Flowever, in the Northern 
and Southern hemispheres, it is different. 
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Current instructional units 



Most educational software for teaching and learning about the Moon phases is focused on viewing the 
Moon from one perceptive- primarily from the Earth’s Northern hemisphere. The limitation of this approach is 
obvious. People do not know what the Moon looks like from the Southern hemisphere relative to the Northern 
hemisphere. Some developers have created three-dimensional educational software to address misconceptions about 
the lunar phases (Bell & Trundle, 2008; Hobson, Trundle, & Sackes, 2010), these programs have failed to 
emphasize lunar observations from both hemispheres. 

Instruction design 

Our approach is based on the idea of global lunar patterns, and the avenues for children and adults to 
intuitively understand, first, the Moon’s shape and orientation at any given time, and, second, the differences of the 
lunar view from a variety of observational positions. 

The authors used the latest instructional technology research, such as Mayer’s principles (Mayer, 2009), 
main assumptions of Cognitive load theory (Sweller, van Merrienboer, & Paas, 1998), etc., to develop interface and 
instructions, and address the issues and proposed goals outlined above. The MOON project contained two units: 
Shape and Orientation. 

The Shape unit includes an introduction to orient learners to think about lunar patterns from different 
hemispheres (See the Figure 1). 



In this module you will see what the Moon looks like from the Northern hemisphere. Use the Scroll Bar and 
the Interactive Lunar Calendar to see the lunar phases and learn their names. Pay attention to shape 
changes, but not lunar orientation. 



Moon from the Northern Hemisphere 




Current phase: 
First Quarter 



Interactive Lunar Calendar 
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Figure 1. 

After the introduction, learners will compare the lunar shape for Northern, Southern, and for both 
hemispheres, reviewing similarities and differences of the Moon’s illumination part. The main goal of this unit is to 
teach the idea that lunar shape is the same in the Northern and Southern hemispheres for a given day, but different 
from day to day (See Figure 2). Lunar orientation patterns are also introduced in this unit to provide a context for a 
more in-depth exploration of lunar orientation in the next unit. 
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Interactive Lunar Calendar 



In this module you will observe the Moon from different places 
within both hemispheres. Click on the map makers and see if the 
Moon looks different from different locations. You can also use 
the Scroll Bar and the Interactive Lunar Calendar to see the 
Moon at different lunar phases. You should notice that the shape 
is the same, but orientation of the Moon is different at locations 
of Northern and Southern Hemispheres. 
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Figure 2. 



The Shape unit also shows the Moon and the Earth from space, thus the authors believe this module will 
teach learners the lunar phases from a space prospective (See Figure 3). 



This module shows you a space prospective on the Earth and the 
Moon. Use the Scroll Bar, the Interactive Lunar Calendar, 'Rotate 
Earth' and ‘Switch Hemisphere' buttons to find out why the Moon 
has different shapes. 

View of the Moon and the North pole of the Earth from space 
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Figure 3. 



The Shape unit uses interactive tools to facilitate learning process. Interactive Lunar Calendar shows the 
current day of the Moon synodic period. A user may click on a particular day and see the Moon shape. Interactive 
Scroll Bar has the similar functionality, but allows to observe non-discrete changes during the Moon synodic period. 



The Orientation unit teaches learners that lunar orientation depends only when observer’s position changes 
on latitude. Therefore, the authors developed several modules with interactive scroll bars to simulate observer’s 
movement on an Earth surface (See Figure 4). 
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An orientation of the Moon depends on your position on the Earth. However, It stays the same If you travel 
only at the same longitude! Therefore, If your latitude has been changed during your travel, you will see that 
the Moon's orientation changes too! To check this idea you may use the scroll bars to travel on the Earth. 



Choose a location, moving the scroll bars View of the Moon from a chosen location 




Figure 4. 

The authors developed three games that will facilitate learning process of lunar shapes and orientation, and 
lunar monthly cycle (See Figure 5). Moreover, after each Shape and Orientation parts learners will have multiple- 
choice questions as assessment tools. 




View of the Moon and the Earth from Nc 



i Hemisphere 



- 




Let's play! The Moon revolves around the Earth in 
29.5 days. People define eight lunar phases m order 
to understand that the Moon goes through the same 
cycle each lunar month. The first phase In the cycle 
is New Moon. Then there is Waxing Crescent phase, 
etc. Fill out the boxes with appropriate Moon 
phases, making a cycle! You may use a support tool 
at the left bottom corner. 

If you want to practice with the other hemisphere, 
please dick 'Switch Hemisphere' button. 

When you finish, hit 'Check!' button. If you would 
like to start again, press 'Reset' button. 



Figure 5. 



The MOON project contains an intuitive interface and help hints. The authors activate learners’ prior 
knowledge (e.g. a person may know about the Moon rotation around the Earth or familiar with lunar shapes), 
connect it with new information (e.g. similarities and differences of Moon’s shape and position, depending on 
observer’s hemisphere and day of Moon synodic period), and build effective schemes in the learners’ memory. The 
information is presented in a few chunks that could be easily processed by individuals, regardless of their 
competence level. Furthermore, all features of the instructional software are user-controlled, thus the person may 
lean in her own pace. 
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The major limitation of the current instruction is not covering a locational pattern (e.g. what is the lunar 
location in night sky from a learner’s courtyard). The authors plan to build this module in the future. 

Evaluation work 

The researchers use formative evaluation as an evaluation = method. During and after the process of the 
MOON project development, to the authors conduct a series of evaluations. These evaluations help the researchers 
improve the MOON project. The authors engaged subject matter experts to review the draft materials during the 
development stage and after final release of the MOON project. Also, one-on-one and small group evaluations 
would be conducted. 
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This paper analyzes problems and issues related to the design of professional learning environments for teachers. 
The first section describes the research study that served as the pilot test for the intentional design process©. The 
second section explains the conceptual and theoretical framework of the design process. 

Section One 

The Research Study 

Abstract 

The purpose of the study was to evaluate the affect of instructional strategies on teachers' attitudes toward initiating 
and sustaining change in their practice. Teachers’ beliefs and assumptions about educational change have been 
identified as a barrier and present a major obstacle to teacher professional development offerings seeking to bridge 
the research-to-practice gap in education. The population for this study was comprised of public school teachers 
enrolled in a professional development course in the Mid-Atlantic region of the United States. A purposeful sample 
of three individual cases was selected from this population. An attempt was made to manage change through the 
design of a course that served as the setting for the problem. The inherent complexity of designing effective teacher 
development offerings capable of meeting both cognitive and affective needs of teachers necessitated the 
development of a new design process. The intentional design process© was created to improve the process of 
designing professional development offerings for teachers. Four sources of qualitative data were used for this study: 
interviews, direct observations, participant-observation, and physical artifacts. The tabular materials in this study 
contained quantitative data. The analysis of the results indicated that the instructional strategies in the professional 
development course were successful in influencing teachers’ attitudes toward initiating and sustaining change in 
their practice. The findings indicate that the design paths of the intentional design process© influenced the 
participants’ attitudes toward change in their practice. Findings will be discussed with the intention of defining a 
design process for professional learning. 



Background 

The design of new approaches to professional development could narrow the research-to-practice gap by 
assisting teachers in ways that encourage the ongoing refinement on teachers’ skills and knowledge. Zhao, Pugh, 
Sheldon, and Byers (2002) report that despite large number of survey studies examining factors influencing 
teachers’ uses of technology, “these types of studies tend to neglect the messy process through which teachers 
struggle to negotiate a foreign and potentially disruptive innovation into their environment” (p.483). Studies on 
professional development offerings in K-12 education tend to approach the problem in three ways. The first group 
of studies identifies barriers to implementation of research-based practices in the classroom (Ertmer, 2005, 1999; 
Spillane, 2002). A second group of studies examines the role that teachers’ assumptions and beliefs play in 
implementing research-based practices (Guskey, 2002; Schlager & Fusco, 2003; Windschitl, 2002; Zhao, Pugh, 
Sheldon, & Byers, 2002). A third group of studies approaches the problem through examination of professional 
development offerings and identifies characteristics of high-quality professional development (Birman, Desimone, 
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Porter, & Garet, 2002; Boyle, Lamprianou, & Boyle, 2005; Howard, McGee, Schwartz, & Purcell, 2000; Ley & 
Young, 2001). These three approaches have been used to examine the research-to-practice gap in education, and the 
results are consistent in indicating that the causes of the problem are complex and the gap will not reduce in size 
until teachers decide to change their practice. 

Participants 

The population for this study was comprised of public school teachers enrolled in a professional 
development course in the Mid-Atlantic region of the United States. A purposeful sample of three individual cases 
was selected in order to develop a deeper understanding of how instructional strategies in a professional 
development course influence teachers’ attitudes. The goal of purposeful sampling is to select cases that are likely 
to be “information rich” with respect to the purpose of the study (Gall, Gall, & Borg, 2003). Gall, Gall, and Borg 
(2003) write “The sample size in qualitative studies is typically small, in fact, the sample size might be a single 
case” (p.165). Hydrangea, Daisy, and Rose volunteered to participate in the research study and were selected for 
individual case studies (Table 1). 



Table 1. Demographic Information for Purposeful Sample of Public School Teachers Enrolled in a Teacher 
Professional Development Course 



Name 


Years of 
Teaching 
Experience 


Current 

Grade 

Level 


Workshops 


: as a Participant 


Workshops as a Presenter 


2005-06 


2004-05 


2003-04 


2005-06 


2004-05 


2003-04 


Rose 


7 


8 


10 


10 


10 


3 


2 


2 


Hydrangea 


21 


6 


12 


12 


12 


4 


3 


5 


Daisy 


23 


8 


11 


8 


7 


10 


8 


8 



Research Questions 

The research questions in this study concentrated on relationships between instructional strategies 
embedded in the core and structural features of the professional development course and change in teachers’ 
attitudes. 

1. How do the instructional strategies in the design of a professional development course affect teachers’ 
attitudes towards initiating change in their practice? 

2. How do the instructional strategies in the design of a professional development course affect teachers’ 
attitudes towards sustaining change in their practice? 

Data Collection 

This study sought to identify instructional strategies in a teacher professional development course that 
influence teachers’ attitudes toward initiating and sustaining change in their practice. The four sources of qualitative 
data used for this study included interviews, direct observations, participant-observation, and physical artifacts. The 
tabular materials in this study contain quantitative data. One source of quantitative data, the Level of Technology 
Implementation (LOTI) questionnaire provides information on teachers’ level of technology use. A second source 
of quantitative data. Hall and Hord’s (2001) Stages of Concern Questionnaire (SoCQ) measures beliefs and 
perceptions and will provide information about participants’ levels of concern about the innovation. The LOTI and 
the SoCQ were administered before the course began and after the course was completed. 

Data Analysis 

The multiple sources of data collected for this study include interviews, direct observations, participant- 
observation, and physical artifacts. The data was organized and prepared for analysis by transcribing interviews and 
sorting or arranging data into different types depending on the sources of information. Next all the data was read to 
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obtain a general sense of the information and to reflect on its overall meaning. Then the detailed analysis began with 
a coding process. Text data was sorted and categories were labeled. The coding process generated a description of 
the setting and people as well as categories or themes for analysis. 

Collecting information using an assortment of sources and methods decreases the risk that conclusions will 
reflect the systematic biases or limitations of a specific source or method and allows the researcher to gain a broader 
understanding of the topic being investigated (Yin, 2003). Two strategies were used to enhance the credibility of the 
findings of this research study. The first strategy used to enhance the credibility of the findings, triangulation, 
involved using multiple and different sources and methods to provide corroborating evidence. The multiple 
methods that have been selected to gather data include case notes, case study documents, tabular materials, and 
narratives. The second strategy used to enhance the credibility of the findings, member checks, involves requesting 
participant’s views of the credibility and the interpretations of the findings. Member checks involve taking data, 
analyses, and interpretations back to the participants so that they can assess the credibility of the account. This 
strategy is considered by Lincoln and Guba (1985) to be the “most credible technique for establishing credibility” 
(p.314). 



Data Summary 

The data was analyzed through use of what Yin (2003) calls “elements of explanations” (p.120). Yin 
(2003) writes “To ‘explain’ a phenomenon is to stipulate a presumed set of causal links about it. ... Because such 
narratives cannot be precise, the better case studies are the ones in which the explanations have reflected some 
theoretically significant propositions” (p. 120). The data was ‘explained’ by stipulating a set of causal links between 
the change in teachers’ attitudes and the process used to design the core and structural features of the course. The 
explanations reflected theoretically significant propositions in the following areas: andragogical assumptions, 
identified by Knowles (1990); attributes of innovations that affect their rate of adoption, Rogers (2003); and core 
and structural features of exemplary professional development offerings identified by Desimone, Porter, Caret, 
Yoon and Birman (2002). 

Conclusions from the Study 

The teacher professional development course developed for this study focused on elements of instruction 
identified by research as essential to the improvement of teacher practice (Barron, Kemker, Harmes, & Kalaydjian, 
2006; Hmelo, Holton, & Kolodner, 2000; Marzano, 1998; Wenglinsky, 2000; Wiggins, 1998). The course 
represented an application of the intentional design process©. The following conclusions can be drawn from the 
significant findings reported in this study: 

• The instructional strategies in the course did affect the participants’ attitudes toward initiating change 
in their practice. 

• The instructional strategies in the course did affect the participants’ attitudes toward sustaining change 
in their practice. 

• The process used to design the professional development offering, the intentional design process©, 
was responsible for the effect of the instructional strategies on the participants’ attitudes toward 
initiating and sustaining change in their practice. 

• Participation in the professional development offering enabled each participant to reduce the gap 
between research and classroom instructional practices in the areas of integration of (a) effective 
instructional strategies, (b) multiple assessment strategies, (c) higher-order thinking skills, and (d) 
technology. 

The inherent complexity of designing effective teacher development offerings capable of meeting both 
cognitive and affective needs of teachers necessitated the development of a new process. The intentional design 
process© offers a new solution to an old problem, the research-to-practice gap in education. The innovative 
intentional design process© synthesizes and extends breakthroughs in learning theory, instructional design theory, 
change strategies, and education research developed during the last years of the twentieth century. 
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Section Two 



Beyond the Study 

After the research study was concluded the intentional design process© was further developed and defined. 
The primary goal of this design process is to create exemplary professional learning environments. It is intended for 
all situations which, though different in content, have comparable common consistent structures that allow you to 
handle them the same way by utilizing the same mental operations. Some of the main beliefs on which this process 
is based include: critical thinking and problem-solving skills, self-directed learning with metacognitive support, and 
attitude development or change. The general strategy employed by the intentional design process© involves 
creation of professional learning environments through simultaneous use of the cognitive design path and the 
affective design path to create a bridge between education research and classroom practice. 

Cognitive Design Path 

The cognitive design path addresses the learners' needs as teachers. Reigeluth and Moore (1999) affirm 
that the cognitive domain of learning "deals with the development of understandings and intellectual abilities and 
skills" (p. 52). Three major practitioner needs are pedagogical knowledge, content knowledge, and technology 
knowledge. The primary purpose of the cognitive design path is to meet the learners' needs as practitioners. 

Koehler and Mishra (2009) define pedagogical knowledge as practitioners’ "deep knowledge about the 
processes and practices or methods of teaching and learning. ... A teacher with deep pedagogical knowledge 
understands how students construct knowledge and acquire skills and how they develop habits of mind and positive 
dispositions toward learning" (p. 64). Content knowledge is defined as "teachers’ knowledge about the subject 
matter to be learned or taught" (Koehler & Mishra, 2009, p. 63). 

Koehler and Mishra 2009 write: 

The definition of technology knowledge used in the TPACK framework is close to that of Fluency of 
Information Technology (FITness), as proposed by the Committee of Information Technology Literacy of 
the National Research Council (NRC, 1999). ... FITness, therefore, requires a deeper, more essential 
understanding and mastery of information technology for information processing, communication, and 
problem solving than does the traditional definition of computer literacy. Acquiring technology knowledge 
in this manner enables a person to accomplish a variety of different tasks using information technology and 
to develop different ways of accomplishing a given task (p.64). 

Use of the cognitive design path to meet the learners’ needs as teachers initiates the change process. 

Initiating Change 

Seel (1997) concludes that in order to describe complex interactions more instructional theory needs to be 
based on extensive relationships among variables. The cognitive design path begins the process with the 
identification of the problem setting, purpose, instructional goal and objectives, and the intentional activation of 
essential content, pedagogical, and technology knowledge required to meet them. The cognitive design path 
intentionally activates the cognitive domain by synthesizing the content, pedagogical, and technology knowledge 
essential for meeting the course goals and selecting an instructional strategy that supports the teaching and learning 
process. The iterative design process enables the power of the selected instructional strategies to remain constant. 
The learners’ needs as teachers are met with this level of design and they have access to research-based knowledge 
(Figure 1). 
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Affective Design Path 

The affective design path addresses the teachers' needs as adult learners. Martin and Reigeluth (1999) 
write, "The affective domain refers to components of affective development focusing on internal changes or 
processes" (p. 486). Smith and Ragan (1999) believe that affective learning outcomes involve attitudes, motivation, 
and values, and the expression of these often involves statements of opinions, beliefs, or an assessment of worth. 

The affective design path embeds andragogical assumptions, effective change strategies, and a triad of reflective 
actions into the core and structural features of the professional learning experience. The purpose of the affective 
design path is to impact practitioners' attitudes toward integrating research-based knowledge into their classroom 
instructional practices. Using the affective design path to meet the teachers’ needs as adults sustains the change in 
their classroom instructional practices. 



Sustaining Change 

The affective design path intentionally creates a link between practice and research through the design of 
the core and structural features of the professional learning offering by synthesizing knowledge related to how adults 
learn, reflective learning, and the diffusion of innovations (Figure 2). This is a deliberate attempt to change the 
behavior, attitudes, beliefs, and assumptions deeply held by teachers. The teachers’ needs as adult learners are met 
with this level of design and they are able to integrate research-based knowledge into pre-existing classroom 
instructional practices. 




produces - 






Figure 2. The affective design path continues the iterative process and sustains educational change. 
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The ADDIE Bridge 



ADDIE is an acronym for the five main steps of course development: Analysis, Design, Development, 
Implementation, and Evaluation. To create a self-determined learning environment you must create a bridge 
between the cognitive and affective design paths that connects the theoretical with the experiential. Use of the 
cognitive design path to create the research-to-practice connection maintains the power of instructional strategies as 
a constant and initiates educational change. Use of the affective design path to create the practice-to-research 
connection increases the power of instructional strategies and sustains educational change. Inserting the ADDIE 
bridge between the core and structural frameworks in the instructional design process© enables the development of 
renewable change in classroom practices (Figure 3). 




Figure 3. Renewable educational change begins when the ADDIE Bridge is inserted into the gap between the core 
and structural framework. 



Renewable Change 

The power of the instructional strategies can be deliberately increased through the intentional design of the 
core and structural features of the professional development environment. The iterative nature of the intentional 
design process© offers multiple pathways for increasing the power of instructional strategies. Following the logic 
that created the intentional design theory© enables another assumption to be made, when the design process 
supports the ADDIE bridge between the cognitive and affective design the process leads to the design of a self- 
directed learning environment linking education research and classroom practice. It is the last step in the process. 
Once self-directed learning is activated, the learning circuit is complete and the learner is released to seek the 
knowledge they need to know. The process continues, but this time at the learner’s discretion not the designers. 

Section Two: Conclusion 

Three types of change have been discussed in relation to the design of professional development programs; 
initiated change, sustainable change, and renewable change. Of the three renewable change is the most difficult to 
achieve when designing professional development programs. You often hear educational planners speak about 
initiating and sustaining educational change, but you rarely hear them mention renewable change. For the purpose 
of this paper renewable change is defined as "to begin again." Using the intentional design process© makes it 
possible for teachers to determine the direction of their next professional learning experience. After finishing the 
course each teacher spoke about how participation in the course had changed them. They also spoke about the effect 
the course had on their deeply held beliefs concerning teaching. 

Hydrangea 

“It has actually changed me entirely. . ..That process influenced my teaching in the best way. That 
definitely will be permanent.” 

Rose 

“I have found it to be a good experience and it is useful to me, I hope I can take those changes and make it 
happen in my classroom. At least it has given me some depth as to my teaching, where I need to grow and make the 
changes 1 need to make." 
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Daisy 

"It’s not a pre-programmed method of teaching. It’s just different and it is better. It's the way I want to 

teach." 

The course designed using the intentional design process© produced change in the each teacher's attitude 
toward change in their classroom instructional practices. They are ready to "begin again." 



References 

Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research on technology in K-12 
schools: Technology as it relates to the National Technology Standards. Journal of Research on 
Technology’ in Education, 35, 489-507. 

Birman, B. F., Desimone, L. M., Porter, A. C., & Garet, M. S. (2000). Designing professional development that 
works. Educational Leadership, 51(8), 28-33. 

Boyle, B., Lamprianou, I., & Boyle, T. (2005). A longitudinal study of teacher change: What makes professional 
development effective? Report of the second year of the study. School Effectiveness & School 
Improvement, 16(7), 1-27. 

Creswell, J. W. (1998). Qualitative inquiry’ and research design: Choosing among the five traditions. Thousand 
Oaks, CA: Sage. 

Ertmer, P. A. (1999). Addressing first-and-second order barriers to change: Strategies for technology integration. 
Educational Technology, 41(4), 47-61. 

Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration. 
Educational Technology’ Research and Development, 53 (4), 25- 39. 

Gall, M. D., Gall, J. P. & Borg, W. R. (2003). Educational research: An introduction (7th ed.). New York: Allyn 
and Bacon. 

Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching, 8(3/4), 381-389. 

Hall, G. E., & Hord, S. M. (2006). Implementing change: Patterns, principles, and potholes. New York: Pearson 
Education, Inc. 

Hmelo, C. E., Holton, D. L., & Kolodner, J. L. (2000). Designing to learn about complex systems. Journal of the 
Learning Sciences, 9(3), 47-298. 

Howard, B. C., McGee, S., Schwartz, N. H. & Purcell, S. (2000). The experience of constructivism: Transforming 
teacher epistemology. Journal of Research on Computing in Education, 32 (4), 455-465. 

Knowles, M. (1990). The adult learner: A neglected species (4th ed.). Houston, TX: Gulf Publishing Company. 

Ley, K., & Young, D. B. (2001). Instructional principles for self regulation. Educational Technology’ Research and 
Development, 49 (2), 93-103. 

Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent 
Regional Educational Laboratory. 

Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic inquiry’. Beverly Hills, CA: Sage Publications. 

Rogers, E.M. (2003). Diffusion of innovations (5 th ed.). New York: Free Press. 

Schlager, M. S. & Fusco, J. (2004). Teacher professional development, technology, and communities of practice: 
Are we putting the cart before the horse? In S. Barab, R. Kling, and J. Gray (Eds.), Designing Virtual 
Communities in the Service of Learning. Cambridge, UK: Cambridge University Press. 

Smith, P.L., & Ragan, T. (1999). Instructional design. New York: John Wiley & Sons. 

Spillane, J. P. (2002). Local theories of teacher change: The pedagogy of district policies and programs. Teachers 
College Record, 104(5), 377-420. 

Wenglinsky, H. (2000). How teaching matters: Bringing the classroom back into discussions of teacher quality. 
Princeton, NJ: Milken Family Foundation and Educational Testing Service. 

Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San 
Francisco: Jossey-Bass. 

Windschitl, M. (2002). Framing constructivism in practice and the negotiation of dilemmas: An analysis of the 
conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational 
Research, 2, 131-175. 

Yin, R. K. (2003). Case study research: Designs and methods. Thousand Oaks, CA: Sage. 

Zhao, Y., Pugh, K., Sheldon, S., & Byers, J. L. (2002). Conditions for class technology innovations. Teachers 
College Record, 104(5), 482-515. 



440 




Using Mobile and Research-Based Technologies in the K-8 Math/Science 
Curriculum: The xSTEM Framework 

Julie Reinhart, jreinhart@sxu.edu 
Jane Lundin, lundin@sxu.edu 

Saint Xavier University, Chicago 



Purpose 

The xSTEM framework was designed to aid teachers with developing hands-on instruction that extends current 
instructional practices by integrating scientific and mobile technologies into K-8 math and science curricula. The 
xSTEM framework was developed by members of the STEM Center team at a private university in a major 
metropolitan area in the Midwestern United States. The STEM center received a federal grant to promote engaging 
instructional practices in science, technology, engineering and mathematics (STEM) in elementary schools with the 
goal of promoting student interest in STEM careers. The STEM Center provides professional development 
opportunities, for teachers, which are based on the xSTEM framework. 

Background 

There are two key elements to successful utilization of the xSTEM framework. The first is providing teachers with 
professional development that affords them the opportunity to actively engage in authentic inquiry-based projects. 
Second, the professional development needs to take place in a technology and resource rich environment. The 
research supporting these two key elements is briefly discussed below. 

Conducting authentic scientific experiences is an area in which most elementary school teachers do not have a 
background in (Akerson, Morrison, & McDuffie, 2006). Thus professional development initiatives can improve 
teachers' ability to provide authentic inquiry-based instructional practices (Akerson, Townsend, Donnelly, Hanson, 
Tira, & White, 2009). In fact, research has found that teachers are more likely to incorporate inquiry-based scientific 
practices in their own K-12 classrooms if they themselves have actively participated in inquiiy -based STEM 
projects, as part of professional development practices (Kazempour, 2009). Furthermore, it has been established that 
inquiry-based projects increase student motivation to learn and improve learning outcomes (Thadani, Cook, Griffis, 
Wise, & Blakey, 2010). 

Additionally, research indicates that the use of authentic scientific/research-based tools is important in elementary 
education in order to foster authentic and deeper scientific learning in young students (Akerson et al, 2009). If 
teachers want to improve their students’ scientific learning with scientific technologies, they themselves need to 
have experience with such technologies (Zhao, Lei, & Frank, 2006). 

The use of mobile technologies provides students with more authentic learning experiences as students are able to 
gather data at the place of study such as playgrounds, fields and so on. This affords students the opportunity to 
engage in place-based science projects, such as playground experiments, soil and natural plant research, which have 
been found to promote deeper and more authentic learning in students (Sarkar & Frazier, 2008). 



Professional Development Using The xSTEM Framework 

Professional development at the STEM center engages teachers in the research process within a resource-rich 
environment. Because teachers are actively engaged, they themselves are able to fully understand the importance of 
the research process and the nuances of the process so they are better able to facilitate the process with their own 
students. Teachers learn about the xSTEM framework, which connects existing instructional practices with more 
innovative, research-based practices. This connection suggests that teachers will be more apt to adapt similar 
projects for their own classroom (Carrier, 2009; Martin, 2003; Zhao et al, 2006). 
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Teachers are encouraged to develop inquiry-based activities using the xSTEM framework that is standards-based yet 
inquiry oriented. Teachers start with their traditional curriculum, which is based on local, state and national 
standards. Teachers then develop an inquiry-based project or activity that begins with an "essential question" for 
which their students will need to develop a research plan and gather data, which they evaluate and analyze as they 
work towards a solution to the “essential question.” As students implement their research plan they make 
observations and evaluate, question, and revise their plan as necessary in order to answer the “essential question” 
that the teacher posed. The inquiry portion of the project leads to the creation of a final product that the students 
develop, which is directly related to the original essential question. 

Roller Coaster Math Example 

For example, in a fifth grade class where students are learning about measurement and estimation, teachers might 
ask students to discuss what attributes of a roller coaster make roller coasters go faster? In traditional math classes 
students might make predictions based on past experiences or by comparing roller coaster images and determining 
which one would go faster. While this practice is acceptable in many regards, by using the xSTEM framework, and 
mobile scientific tools, teachers can have students actively engaged in answering the question “what makes roller 
coasters go faster?” by using mobile motion sensors to gather data. 

For instance, students can track the speed of a toy car that rolls down a plank in order to simulate a steep decline on 
a roller coaster. Students can change the slope of the plank, length of the run, and so on based on their research 
plan. Students can make multiple observations in order to determine which set-up will make the toy car go faster. 
The motion sensor tracks each run and collects data that students can then analyze to make their final determination 
of what makes the toy car go the fastest. 

There are many more possibilities for using the motion sensors with the roller coaster math activity for example: 
students can test out multiple playground slides to determine which one allows students to go faster. They can 
determine what quantifiable aspects of the slide influence the speed of the individual going down the slide. The 
mobile motion sensor can be used to track the speed of the individual going down the different slides. 

Finally, once students determine what influences the speed of a roller coaster, students can go one-step further and 
design their own roller coaster by applying what they learned from the inquiry-based lesson. This roller coaster can 
be something they design on a piece of graph paper, construct using a computer simulation, or even create a fully 
operational roller coaster such as the K’nex ™ roller coaster construction materials. 

Conclusion 

As active participants in a resource-rich environment, teachers are given the opportunity to determine which 
technologies and techniques might work best for their unique school environment. Teachers can decide if a 
technology would be a viable option for their own classroom by interacting with scientific technology they might 
not otherwise have access to in their own classrooms. When teachers have time to experiment and become familiar 
with different technologies (scientific and mobile), teachers are more apt to use such technologies in their own 
classroom because they gain a greater familiarity and have a better understanding of the technologies (Zhao et al, 
2006). When the opportunity to experiment with technology is coupled with the opportunity to create inquiry-based 
STEM projects teachers are better able to integrate inquiry-based STEM-related projects into their classroom. 
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Background 

There is a growing appreciation in the research literature for studies that address what instructional designers do 
in everyday life and how they do it. For example, Visscher-Voerman and Gustafson (2004) studied the design strategies of 
professional instructional designers to determine how then activities were similar and different from ADDIE, and 
instructional design process model. Williams, South, Yanchar, Wilson, and Allen (2011) investigated how instructional 
designers use evaluation to improve their instructional products. Williams et al. commend studies that have given a clearer 
picture of how instructional designers do their work. They suggest these studies shed light on “lessons learned” from 
practice and note how important this information is to improve the scholarly research in this field as well as how we 
educate and train instructional designers. However, sometimes lessons learned, principles, or heuristics resulting from such 
studies lack nuances of design that could be beneficial to other designers. 

Boling (2011) offers a different perspective to disseminating design knowledge - specifically, how designers 
share precedent through design cases (Boling, 2011). Precedent refers to the “experiential or episodic knowledge” that is 
“often not explicit but held in minds rather than recorded in an organized manner and is accumulated through practice 
rather than through instruction” (Lawson, 2004, p. 453). Boling differentiates design cases from other forms of research 
including design and development research (Richey & Klein, 2007), formative research (Reigeluth & Frick, 1999) and 
design-based research (Design-based Research Collective, 2003). Described as a “distinct form of knowledge 
dissemination,” design cases use precedent in a “proactive versus reactive. . .synthetic versus linear. . .concrete and situated 
versus theoretical and generalized. . .and fluid versus fixed” manner (Boling, p. 4). In design cases, precedent is not used as 
explicit guidance but rather as “a contribution to that designer’s store of patterns, her ability to discern and appreciate the 
qualities of patterns, and her acts of innovation both in the near and long term” (Boling & Howard, 2011). 

Design cases can be valued for their precedent, utility, and rigor. While arguing there is value in design cases that 
have utility but do not necessarily have rigor. Smith (2010) suggests the more rigorous the design case, the more likely the 
case will serve as a useful case in a broader context. Smith suggests a naturalistic paradigm (Lincoln & Guba, 1985) and 
action research (Argyris, Putnam, & Smith, 1985) serve as good starting points for organizing, conducting, describing, and 
reporting a design case. Standards of rigor will develop over time through the interplay of those who develop design cases 
and their peers who review them. 



A Reflection Paper 

The aim of the reflection paper was to further explore the nature of design cases and how to establish rigor in 
such cases. The idea for the presentation was to explore these issues within the context of an instructional design project. 
This project is the redesign of an online doctoral course. While it is not necessary to base design cases on specific 
instructional design theories and models (Boling & Howard, 2011), the use of first principles of instruction (Merrill, 2002) 
was selected as a framework to redesign the course. The aim of the reflection presentation was not to present the design 
case (yet) but rather to generate discussion about design cases in general and seek feedback about this particular case. The 
aim of the session was to discuss: 

• the process and value of producing and publishing design cases - specifically, how design cases differ from other 
research designs in the field of instructional design and development, 

• how to set up a design case, 

• how to establish rigor through the use of existing research paradigms and methods, and 

• ideas for presenting this particular case as a design case. 
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The Proposed Design Case 



Since submission of the conference proceedings preceded the reflection discussion, the information 
provided here describes the case and presents initial design decisions pertaining to each of the five principles. 
Discussion at the conference will further inform how the design of the course progresses and is presented as a design 
case. 

I teach a directed research course in instructional design and development as part of a doctoral computing 
technology in education curriculum. Our doctoral program is a limited-residency program where students come to 
campus twice per year for an extended weekend during coursework. Once students complete their 700-level 
coursework, they are not required to return to the physical campus. Therefore, all of the student-to-faculty, student- 
to-student, and student-to-resource interaction happens virtually via Blackboard (including email, web conferencing, 
and discussion forums) and either phone or Skype conversations. 

The program’s curriculum includes 700-level courses in assistive technology, human-computer interaction, 
instruction delivery systems, instructional design and development, knowledge management, learning theory, online 
learning environments, online program administration, telecommunications, and qualitative and quantitative 
research methods. Students are required to take 64 credit hours, of which 40 are for courses and 24 are for the 
dissertation. Of the 40 credit hours in coursework, students are required to take two sections of an 800-level directed 
research course (4 credits each) in their subject area of interest (e.g., assistive technology, human-computer 
interaction, instructional delivery systems, instructional design and development) prior to entering candidacy. The 
directed research course is intended to help students focus on a specific research problem within a specific content 
area that a faculty member can support. With the professor’s permission, students can take both 800-level courses 
with the same professor; however, many students prefer to take their two 800-level courses with two different 
professors. 



Problem 

During the first few years of graduate programs, students are busy learning about particular concepts, 
principles, theories, practices, and techniques their chosen field of study (Lovitts, 2007). Boote and Beile (2005) 
emphasize the need to develop skills needed to analyze and synthesize research in a field of specialization “is 
crucial to understanding educational ideas” and “Such preparation is prerequisite to choosing a productive 
dissertation topic and appropriating fruitful methods of data collection and analysis” (p. 3). Unfortunately, many 
students enter the dissertation phase of their studies unprepared for the transition from course-taker (a consumer of 
knowledge) to independent researcher (a producer of knowledge) (Lovitts, 2005). 

In the past, the 800-level courses were primarily project courses that extended the work of the 700-level 
courses and did not necessarily help students become independent researchers. For example, the 700-level 
instructional design course might have the students design and develop an instructional product and the 800-level 
course (now the directed research course) would be a continuation of that design with the implementation and 
evaluation of the product. While the school’s faculty has decided to make the 800-level courses research-focused 
with the goal of helping students transition from coursework to dissertation, and they have subsequently changed the 
nature of their 800-level courses to reflect that shift, faculty have autonomy in course design and delivery. That is, 
there is no standard 800-level course template. 



Purpose 

The purpose of this design case will be to describe how I redesigned the course and the decisions I made 
along the way as I attempted to apply first principles of instruction (Merrill, 2002a) as a guiding framework. First 
principles “describe a cycle of instructional phases consisting of activation, demonstration, application, and 
integration all in the context of real world problems or tasks” (Merrill, 2007, p. 6) and offer a structured and 
prescriptive approach toward instruction. This first phase of the redesign is to think about how first principles could 
be applied to facilitate instruction towards achieving the course’s learning outcomes. As a result of the course 
redesign, it is anticipated students will be more prepared to conduct independent research; the instructor will have 
clearer direction about specific instructional activities that can guide students in achieving the learning outcomes; 
and the instructional designer will have a basis for evaluating the course design and instructor implementation of this 
course. It is also hoped that once the design case is completed, readers can gain insight into how they might design 
similar courses. 
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Current State 



Currently, the goal of my directed research course is for students to conduct a review of the literature about a 
topic of interest in the field of instructional design. There are three major assignments including: 1) literature review topic 
proposal, 2) annotated bibliography, and 3) literature review. The course is designed primarily as an independent study 
and there is minimal guided instruction. Detailed assignment guidelines are provided, students are expected to complete 
the assignments independently, and I provide copious feedback once the assignment has been submitted. Students have 
one chance to submit the assignment for a grade. After teaching a few iterations of the course in this manner, I have found 
that students need more guidance and formative feedback on the front-end of the research process - particularly with the 
identification of a research-worthy problem and how to use that problem as the basis of a dissertation idea paper. While 
students study research problems, questions, and methods in their research methods courses, these elements are included 
as part of a larger focus on quantitative or qualitative analysis and reporting. Also, depending on when during the program 
the student decides to take their methods courses, these front-end research components are often studied out of the context 
of the students’ specific research interests. Students would benefit from thinking about research problems, goals, 
questions, and methods within the context of an authentic task that could potentially become their dissertation research. In 
addition, implementing structured and guided instruction with more frequent, yet smaller assignments, would allow more 
opportunities for practice and formative feedback throughout the 16-week online course. Kirschner, Sweller, and Clark 
(2010) recommend strong and direct guidance for both novice and intermediate learners and suggest a minimally-guided 
approach may be less effective and even produce negative results if students “acquire misconceptions or incomplete 
disorganized knowledge” (p. 84). 



Desired State 

Rather than focusing on a review of the literature as the final deliverable of the course, I want to design this 
course so that there is more focus on the beginning stages of a dissertation study such as identifying a research 
problem, goal, questions, and approach. That is, I want students’ work in this course to align more closely to what is 
expected during the initial stages of the dissertation. Reviewing the literature remains a critical component of the 
course; however, the focus of the instruction is less on the literature review process and more on the front-end 
components of an independent research study and the process of writing an idea paper. It is hoped that by aligning 
this course with the expectations of the first phase of the dissertation process (i.e., development of the idea paper), 
students will develop their research knowledge and skills in the context of an authentic task, gain a clearer 
understanding of the expectations of a dissertation, and potentially be able to use the draft idea paper they prepared 
in the course as a starting point for their dissertation study. 

Merrill’s First Principles 

Reigeluth and Carr-Chellman (2009) suggest that research in instructional theory has reached the stage 
where it is time to develop a common knowledge base. They argue that developing consistent terminology, for 
example, would encourage and advance future research in this area. Specific to developing a common knowledge 
base related to instructional-design theories and models, Merrill (2002a) set out to identify a set of common 
principles about instruction. Specifically, he proposes a set of “interrelated instructional design principles” referred 
to as First Principles of Instruction and suggests that the “effectiveness, efficiency, and engagement of a particular 
model or method of instruction is a function of the degree to which these principles are implemented” (Merrill, in 
Reigeluth & Carr-Chellman, p. 43). To that end, he challenges researchers to verify these principles in a “wide 
variety of settings, for a wide variety of different audiences, in other cultures, and across subject-matter domains” (p. 
56). 

In an attempt to verify these principles in the context of an online graduate research course, I am using 
Merrill’s (2002a) first principles of instruction to guide the redesign of this course. Merrill (2002b) defines a first 
principle of instruction as “a prescriptive design principle on which various instructional design theories and models 
are in essential agreement” (p. 42). These principles are underlying, basic methods that support more specific 
instructional activities. Merrill suggests learning is promoted when: 

1 . Learners engage in solving real-world problems 

2. Existing knowledge is activated as the basis for new knowledge 

3. A demonstration of the new knowledge is given 

4. The learner applies the new knowledge 

5. The learner integrates knowledge into his or her world 
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Evaluation of Course Design and Teaching and Learning Quality 

Flick, Chadha, Watson, Green, and Zlatkovska (2009) address the first principles from a student evaluation 
perspective. They conducted an empirical study to verify these principles by asking students about their learning 
experiences in undergraduate, face-to-face, college courses that they had taken or were about to complete. That is, 
Frick et al. (2009) focused on students’ perceptions of teaching and learning quality (ends) rather than the design 
process (means). In a subsequent study, Frick, Chadha, Watson, Green, and Zlatkovska (2010), propose an 
instalment to measure teaching and learning quality (TALQ). A component of the instalment is a measurement of 
evidence of first principles. While the results of these studies provide guidance to instructors for measuring the 
principles in their online courses, the authors suggest further research is needed to determine whether 1) students 
learn better or learn more when instructors use first principles (Frick, et al., 2009) and 2) the TALQ instalment is 
useful in providing feedback to instructors regarding how effectively they designed and implemented their courses 
according to the first principles (Frick, et al., 2010). In order to determine whether students learn better or learn 
more, or whether courses have been effectively designed according to the first principles in a fully online learning 
environment, it is important to understand how to design a course that effectively incorporates these first principles 
in this setting. 

First Principles Applied to an Online Directed Research in Instructional Design and Development 

Course Description 

Students will work independently and under the guidance of the professor to produce a complete draft of an 
idea paper. The idea paper is the first dissertation deliverable and includes the following major components: 
background/introduction, problem statement, dissertation goal, research questions, relevance and significance, 
barriers and issues, brief review of literature, approach, milestones, resources, and references. Through direct 
instaiction, activities, assignments, and practice, students will hone their research focus in the field of instaictional 
design and development and gain knowledge and skills related to the idea paper process and product. 

Course Goal and Objectives 

The goal is to identify and develop the fundamental components of a dissertation idea/concept paper. Students will: 

• Identify a research topic in instructional design and define its component issues. 

• Identify a research-worthy problem within the selected topic area. 

• Define the goal of the research study and formulate research questions. 

• Describe the relevance and significance and barriers and issues relating to the proposed research. 

• Analyze, synthesize, and evaluate the research literature and present a brief review of the literature that 

supports the research problem and goal. 

• Formulate a research approach that is suitable for the proposed research. 

• Outline milestones and resources for the proposed research. 

In the following sections, each of the five principles and its corollaries are briefly described along with 
guiding questions for course design and an initial exploration of how the principles could be used to guide the 
redesign of the directed research course. 

Principle 1 - Problem-Centered (corollaries: show task, task level, and problem progression) 

Merrill (2007) uses the term, problem, to represent a real-world whole task as opposed to various parts of a 
task that are similar to what one might encounter in the real-world. He contrasts problem-centered (aka task- 
centered) instaiction with topic-centered instaiction where problem-centered instaiction addresses the whole task 
and topic-centered instaiction addresses components of a task and each task is taught separately. Guiding questions 
pertaining to course design include: What is the whole task or problem? What are the inputs, goal, and solution for 
the whole task and how can the whole task be represented as a problem-solving process within the course? 

Corollary - Show Task: Merrill (2002a) suggests that instead of using objectives to gain attention and 
introduce the instruction, a better orientation to the instaiction includes a demonstration of the whole task that is 
similar to the one being taught. This demonstration would preferably show something similar to what the learners 
would be able to do after they complete the instruction. 

In this instance, the goal of the course is for students to write a complete rough draft of a dissertation idea 
paper. At the beginning of the course, various exemplar research papers in the field of instaictional design will be 
presented. (Since it is not customary for the school to show students examples of other students’ idea papers, 
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published research papers or dissertations will be selected instead.) Each example will highlight components of the 
study that are included in the idea paper (i.e., background, research problem, goal, research questions, and 
approach). It is important to select exemplars that represent the broad spectrum of research being done in this field 
(e.g., quantitative, qualitative, mixed-method, design-based, formative, and evaluative studies). Showing these 
various examples would emphasize the various ways to address research problems in the field of instructional 
design. 

Corollary - Task Level: Merrill (2002a) identifies four levels on which the problem should be addressed 
including: “(a) the problem, (b) the tasks required to solve the problem, (c) the operations that comprise the tasks, 
and (d) the actions that comprise the operations” (p. 46). In this instance, the problem or whole task is producing a 
dissertation idea paper. The tasks include: identifying a research-worthy problem, goal, and research questions; 
describing the relevance, significance, barriers, and issues; conducting a brief review of the literature, and outlining 
an initial approach (i.e., research method). The operations and actions for completing these tasks are taught in the 
700-level research methods coursework are will be reviewed as part of this course. For the first and second 
assignment, students will be asked to identify two recent scholarly research articles in the field of instructional 
design and using a rubric similar to the CASP tool used to appraise qualitative research (Public Health Resource 
Unit 2006, see: http://www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdf), 
appraise the articles by identifying the various components of the idea paper (e.g., problem statement, goal, research 
questions, etc.). Specific questions in the rubric will assist the learner in identifying strengths and weaknesses of 
each of these components. We will focus only from the problem identification to the methodology (as opposed to 
evaluating the entire research article) since our aim is to help students write the idea paper, which includes from 
problem identification through a brief outline of the approach. 

Corollary - Problem Progression: Merrill (2002a) advises some problems can be quite complex. For 
example, writing an idea paper is considered by many dissertation students one of the most difficult hurdles to 
overcome. Merrill suggests providing simple problems at the beginning and once those problems are mastered, 
moving to more complex problems. It is difficult to conceptualize an instantiation of problem progression for this 
course because there is no simple representation of an idea paper (i.e., the whole task). Therefore, the following 
modification is provided to address the idea of presenting assignments that progress from simple to more complex; 
however, each does not represent the whole task but rather offers a scaffold to help the student master the whole 
task. 

For example, students are presented with exemplar idea papers and are shown components of the papers 
that make them good. Next, students are instructed to identify two recent scholarly research articles and appraise 
them on the quality and rigor of the study. Following these two appraisals, students will be instructed to produce a 5- 
7 page “abridged idea paper” as the third assignment. This paper includes the following components: 

• Title page (include your name, and working title of your proposed research) (1 page) 

• Relevant expertise and experience (brief overview of your relevant expertise and experience in the selected 

research area) (1/2 page) 

• Research problem (1-2 pages) 

• Research questions (1/2 page) 

• Initial concept for solving the problem (1-2 pages) 

• References (APA format) (1 page) 

Students are given formative feedback on viability of their research problem and proposed solution. Giving 
feedback on this abridged version of the idea paper will allow the professor to provide feedback earlier in the 
process and provide remediation, if necessary before the student spends too much time on an idea that is not viable. 
Students whose research ideas are not viable continue to work on this assignment and receive feedback until a viable 
research idea is presented. The question remains as to whether to grade this particular aspect of the course (i.e., 
summative vs. formative feedback). Perhaps the student is give three chances with formative feedback to produce a 
viable idea before a summative grade is assigned. Another option is to assign a pass/fail grade for this course instead 
of grading each assignment. 

Principle 2 - Activation (corollaries: previous experience, new experience, and structure) 

Activation pertains to the idea that rather than jump right into instruction, it is important to prime the 
student by anchoring the instruction to previous experiences, making the instruction relevant so that it can be used to 
build new knowledge, and providing a schema or asking students to recall a schema for them to organize new 
knowledge (Merrill, 2002a). Guiding questions include: What are some ways to activate relevant previous 
knowledge pertaining to the whole task of writing an idea paper? 
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In this instance, students can employ the same rubric they used to appraise research articles to appraise 
their components of their idea paper. In order to help students and provide structure as students prepare their 
abridged version of the idea paper, a worksheet can be used to guide them through the process (Ellis, 2006, see 
http://scis.nova.edu/~ellist/Research/IdeaPaperIdeas.pdf for an example.). 

Principle 3 - Demonstration (corollaries: demonstration consistency, learner guidance, and relevant media) 

Merrill (2002a) stresses the importance of not only presenting information (e.g., “These are the components 
that need to be in an idea paper.”) but also portraying the information to be learned (e.g., “Here is an example of a 
complete idea paper that includes the various components.”). Portraying information means the information is 
demonstrated as a part of a specific situation or case. Specific portrayals of information help the learner to remember 
and apply the information. Guiding questions include: What strategies can be employed to demonstrate effectively 
the whole task? What are some ways to guide students? What media can be incorporated that is relevant to the 
course? Again, in this case, it would be worthwhile to show examples of completed idea papers. If these examples 
are unavailable, an alternative might be to show exemplar articles and identify the various components that would be 
included in an idea paper. 

Corollary - Demonstration Consistency - The corollary, demonstration consistency, refers to the need to 
ensure that the instructional strategy is consistent with the content category and learning outcome. Various content 
categories include facts, concepts, principles, rules, and procedures (Morrison, Ross, Kalman, & Kemp, 2011). For 
example, if one is teaching the concept of a problem statement, a definition is provided followed by a best example 
of a problem statement. To help students identify problem statements, examples and non-examples could be 
provided and students could differentiate between good and bad problem statements within the context of their 
research topic. Once they are able to identify a problem statement, they could be given the opportunity to practice 
writing their own problem statements. Perhaps this is an aspect of the course that could be integrated into an online 
discussion forum and not necessarily part of a course assignment. 

Corollary - Learner Guidance - Merrill (2002a) advises that learner guidance should be provided early an 
often especially at the early stages of instruction. Explicit guidance focuses the learner’s attention on the relevant 
information to be learned. In this course, formative feedback is provided on every assignment via Microsoft Word’s 
revision marking mode. Comments are provided to guide the student as the write their idea paper. For example, 
students who miss the mark in identification of a research-worthy problem are given formative feedback to guide 
them in the appropriate direction. This feedback might include guiding questions, comments, and references to 
additional information that will help them improve their work. 

Providing multiple representations of the content being taught along with demonstrations of the content is 
important. Multiple representations of ill-defined problems (e.g., providing various types of approaches to solving 
research problems in the field of instructional design) will help the learner realize that there are many ways to 
solving research problems. These multiple representations will result in the learners having a broader perspective of 
how they might solve their specific research problem. 

Corollary - Relevant Media - Merrill (2002a) advises, “Learning is promoted when media play a relevant 
instructional role and multiple forms of media do not compete for the attention of the learner” (p. 48). In this 
instance of an online course, the incorporation of relevant media is critical. Relevant videos, for example, can 
support the various components of the idea paper from writing the research problem to conducting the review of the 
literature. These videos will be incorporated into the course as supplemental reference material for the students. 
Additional media including audio and video feedback can be embedded into student assignments to provide 
clarification over the traditional text-based feedback. 

Principle 4 - Application (corollaries: practice consistency, diminishing coaching, varied problems) 

This principle refers to the idea that learning is promoted when students are required to apply what they 
have learned. Guiding questions include: What types of assignments can be given to students that enable them to 
practice the whole task? What feedback strategies can be used (e.g., rubrics, in-text comments, audio feedback, 
video feedback)? 

The corollary, “practice consistency” pertains to the idea that learning is promoted when the application of 
the content is consistent with the stated learning objective (Merrill, 2002a). In this instance, one of the learning 
objectives is: identify a research topic in the field of instructional design and development. Following a presentation 
about research in instructional design and development, learners apply what they learned by writing a paper that 
includes the identification of a research topic that is relevant and timely. Two additional corollaries, diminishing 
coaching and varied problems are also described. Merrill (2002a) suggests learning is promoted when appropriate 
guidance and feedback is provided up front and gradually withdrawn. Learning is also promoted when a sequence of 
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varied problems is presented and learners are required to solve them. In this instance, this corollary can be 
represented through the appraisal of a variety of research articles in the field of instructional design. 

Principle 5 - Integration (corollaries: watch me, reflection, and creation) 

Learners integrate instruction when they can show evidence that they have gained new knowledge and 
skills. This evidence can be demonstrated when learners demonstrate improvement of a skill or can show how their 
new knowledge is used in the real world (Merrill, 2002a). Guiding questions include: How can we incorporate 
opportunities for learners to publicly demonstrate their new knowledge and reflect on their learning? 

In this instance, students submit a complete first draft of an idea paper at the conclusion of the course. Students 
receive feedback based on a rubric that is used for evaluating dissertation idea papers. Following the course, students 
can be invited to present their research idea, defend their newly acquired knowledge, and receive feedback from 
faculty and peers at an on-site or virtual poster session. 



Next Steps 

The purpose of the reflections presentation was two-fold: to explore the nature of design cases and how to 
establish rigor in such cases and to describe and reflect on the design of an online graduate research course. This 
first phase involved thinking thorough how Merrill’s (2002a) first principles of instruction might be used to guide 
the design of the course. The initial ideas are presented here. The next steps include: 1) completing the course 
redesign and development, 2) implementing the design 3) reporting the instantiation of the design including 
important design decisions, mistakes made throughout the design process, and vicarious experiences about the 
design and implementation process, and 4) evaluating the results of the course design and teaching and learning 
quality. Frick, et al.’s (2010) TALQ instrument will be used in two ways: first, to determine whether students learn 
more or better as a result of the implementation of first principles and second, to determine whether the instrument is 
useful in providing feedback regarding how effectively the course was designed and implemented according to the 
first principles. 
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Abstract 

A purposive sample of six teachers, grades four through 12, in public and private schools, participated in a study that 
explores their use of project-based learning (PBL) in the classrooms. Four major uses emerged: initiation of 
learning, extension of learning, reinforcement of learning, strategic navigation. Interpretations and implications of 
the findings are also presented. 



Introduction 

Project-based learning is an instructional model that is based in the constructivist approach to learning. 
Some key concepts of constructivism are discovery learning, zone of proximal development, scaffolding, cognitive 
apprenticeship, coaching, context, collaborative learning, and the nature of assessment (Duffy & Cunningham, 
2005). Not only does constructivism aim at the construction of knowledge, but it also entails the construction of 
knowledge with multiple perspectives and within a social activity, it is context dependent, and it allows for self- 
awareness of learning and knowing (Duffy & Cunningham, 2005). Project-based learning is one instructional 
model, among others, that encompasses all these aspects of constructivism. 

Advantages of PBL 

As a constructivist model, (PBL) targets the building of the learner as a whole and not simply a model that 
aims at increased knowledge about specific content areas. Katz and Chard (1992) posit that PBL achieves four 
goals: acquisition of knowledge, acquisition of skills, dispositions, and feelings. Through PBL, students become 
intrinsically motivated, more focused, and they develop a range of abilities and skills (Griva, Semoglou, & Geladari, 
2010; Wolk, 1994; Wurdinger, Haar, Hugg, & Bezon, 2007). 

Teachers find many advantages to PBL. They believe that it enables them to teach skills beyond the 
content, making learning more personalized and more varied, and it enables them to teach academic content more 
effectively (Ravitz, 2008). In addition, they perceive PBL to build students’ creative thinking skills as they acquire a 
deeper level of learning and understanding of the subject-matter (Akinoglu, 2008). 

Studies show positive effects of PBL on students’ learning outcomes (Akinoglu, 2008; Chu, Tse, & Chow, 
2011; Griva, Semoglou, & Geladari, 2010). Students are able to move from novices to experts in the domain of 
knowledge (Grant & Branch, 2005). Their critical thinking abilities, their presentation skills, their communication 
skills, and their ability to work effectively on a team are also enhanced (Neo & Neo, 2009; Wurdinger et al., 2007). 
Additionally, students value the fact that their projects are situated in real-life contexts (Gubacs, 2004). 

Challenges of PBL 

Using PBL in the classroom does impose certain challenges on the teachers who use it. 

Taking on the constructivist approach. Thomas (2000) reports that one important challenge is the 
conflict PBL brings to the deep-seated beliefs of teachers in their approach to teaching and the degree of balance 
needed between student control and teacher control over the activities. As teachers get introduced to PBL, they tend 
initially to rely on the transmission of knowledge approach (Blumenfeld, Krajick, Marx, & Soloway, 1994). They 
need time to transition towards the constructivist approach of PBL, whether it is in shaipening their skills or 
changing their beliefs. Teachers must be able to recognize and accept a shift in their function and become 
comfortable with implementing student-centered pedagogies, such as PBL (Grant & Hill, 2006; Rogers, 2010). 

Additionally, teachers must also be able to tolerate the ambiguity and flexibility of the dynamic 
environment created by the student-centered approach. Teachers may doubt their ability to complete the required 
curriculum because of the time needed to spend on projects. In addition, the teachers may be concerned of losing 
control over the topic and the behavior of the students, which might prevent them from allowing students to work in 
small groups. Therefore, teachers have difficulty giving their students the time needed to build their skills; instead, 
they feel that they need to teach the students the skills before starting the project (Hertzog, 2007). On the other hand. 
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Ravitz (2003) posits that even when teachers show enthusiasm about the constructivist teaching approach after 
participating in professional development workshops, they might not find it easy to implement it in their classrooms. 

Curriculum and selection of topic. Another challenge facing teachers is the creation of a balance between 
district curriculum, testing policies, and the large content that needs to be covered within a fixed schedule (Krajcik, 
Blummenfeld, Marx, & Soloway, 1994; Snyder & Snyder, 2008). Moreover, selecting meaningful project topics 
that engage students could also be challenging (Akinoglu, 2008; Howard, 2002; Wirdinger et al., 2007). On the 
other hand, teachers may not have enough expertise on the subject they are teaching to be able to coach the 
investigation properly or students might explore areas that are not necessarily familiar to the teachers (Grant & Hill, 
2006; Howard, 2002). Therefore, once teachers hone their skills in PBL and become creative in planning the related 
activities, they will overcome their concerns of not following the curriculum strictly and choosing the appropriate 
topic (Wirdinger et ah, 2007) 

Management and design in project-based learning. The successful orchestration of all the features of 
PBL is one more challenge teachers face when they carry out PBL activities (Krajcik, Blummenfeld, Marx, & 
Soloway, 1994; Snyder & Snyder, 2008). Teachers need to orchestrate several elements in order to manage a 
project: a variety of resources, information sources, learning contexts, participants, time, tasks, and arrangements. 
This is in addition to planning, monitoring, scaffolding, adjusting, and troubleshooting strategies (Thomas 
&Mergendoller, 2000). Kolodner et ah (2003) discuss the ability of the teacher to manage projects in a large 
classroom, all the while maintaining the engagement of all students, in addition to maintaining a balance between 
the investigative aspect of the project and the interpretation and reflective activities. Moreover, teachers may be 
challenged by their inexperience of designing adequate project-based activities or by their lack of training in critical 
thinking methodology (Akinuglo, 2008; Snyder & Snyder, 2008). Therefore, teachers working in PBL need to gain 
skills in managing environments that are not stagnant, that draw on several resources, and that guide the learners in 
inquiry as much as partner with them in choices. 

Assessing project-based learning. Assessing student achievement in PBL is an additional challenge that 
teachers must address. Marx, Blummenfeld, Krajcik, and Soloway, (1997) state that, in some instances, teachers ask 
students to produce artifacts that do not require the use of critical thinking and assessing these artifacts does not 
measure understanding. They add that assessing artifacts quality is difficult because of the several features that must 
be taken into account, such as design, organization, and accuracy. Additionally, Grant and Hill (2006) argue that 
assessment should include several learning products and not only the final artifact. They suggest portfolios as a 
learning product where students reflect on their learning experience as they go through the phases of the project. 
Similarly, Barron and Hammond (2010) stress the importance of formative assessment and suggest rubrics, solution 
reviews, whole class discussion, performance assessment, written journals, portfolios, weekly reports, and self- 
assessment as other forms of assessment. Moreover, Grant (2011) posits that teachers need to provide their students 
with clear expectations about the project requirements for better assessment. Therefore, teachers need to look at 
assessment in PBL as multifaceted. It targets individual and group performance, concrete products and cognitive and 
metacognitive skills, as well as learning and social skills. 

The nature of collaboration. The collaborative work needed in PBL is one of its most difficult aspects 
(Kapp, 2009). It is essential for teachers to create a classroom culture of collaboration, where students feel 
responsible of helping each other, and of iteration, where they expect to make mistakes in order to learn from them 
(Kolodner et al., 2003). Also, it is important to create a classroom environment that supports mastery and develop a 
constructive view of error, especially since students might defeat the learning goals of the project if they are worried 
about failing more than succeeding (Meyer, Turner, & Spencer, 1997). 

Teachers perceive that PBL can bring many advantages to the learning experience of the students; however, 
as they implement it in the classroom, they may face certain challenges and they need to adjust their teaching 
accordingly. 



Methodology 

The purpose of this study was to investigate how school teachers implement project-based (PBL) learning 
in their classes. 

The research questions were as follows 

1 . How do teachers define project-based learning? 

2. How do teachers choose to use project-based learning? 

3. How do teachers use technology to support their project-based activity? 
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Design 

A case study approach was used to answer the research questions. The unit of analysis was teachers in the 
bounded system of activity of the implementation of PBL. A collective (or multiple case) approach was adopted where 
several teachers were selected in order to get a broader view on the topic (Creswell, 2007). in addition, having multiple 
cases allowed for finding particularities of cases and also common features between them (Stake, 2003). 

Participants 

Six teachers were selected for the exploration of the research questions. The criteria for selection were as 

follows: 

1. Teachers had to be involved in project-based learning for more than one school year. 

2. Projects involved some form of technology integration. 

3. Teachers were willing to participate in the study. 

Personal contacts were used to identify teachers who satisfied the selection criteria. Teachers were 
contacted by email to introduce the research topic and set the appointments for the interviews. The resulting sample 
of teachers covered grades 4 through 12, included four females and two males, and represented three public schools 
and one private school. The teachers selected ensured variation in the sample between gender, types of schools, and 
grade level. Following is a description of each teacher. 

Greg. Greg is a Caucasian middle-aged teacher who teaches 12 th grade English in a public school, in a 
relatively small city. He has been teaching for eight years and using project-based learning with his students for six 
years. His class size is big, comprising thirty students. 

Audrey. Audrey is a young African American teacher who teaches 9 th grade English in an inner city 
school. She has been teaching for two years and doing project-based learning for a year and a half. Audrey teaches 
a special course in English targeting struggling readers. Her class size is small, comprising twelve students that she 
divides into different activity groups. 

Diane. Diane is a young Caucasian teacher who teaches 5 th grade Math and Science in a campus school in 
the city. She has been teaching for five years and using project-based learning all through her teaching career. Her 
class size is medium, comprising twenty-four students. 

Martha. Martha is a middle-age Caucasian teacher who teaches 5 th grade Social Studies in a private 
school in the outer city. She has been teaching for twenty-two years at the 4 th grade level and has been using project- 
based learning all through her teaching career. Her class size is relatively small comprising seventeen students. 

Brenda. Brenda is a young Caucasian teacher who teaches 4 th grade Reading, Math and Social Studies in a 
private school in the outer city. She has been teaching for four years at the 4 th grade level and has been using project- 
based learning for two years. Her class size is relatively small comprising fourteen students. 

Scott. Scot is a middle age Caucasian male who teaches 6 th grade Math and Science, in a public campus 
school in the city. He has been teaching for eight years and he has been using project-based learning all through his 
teaching career. His class size is medium ranging between 18-28 students. 

Data Collection 

Two data collection methods were used. 

Interviews. A semi-structured individual interview was carried out with each of the participants. Each 
interview lasted between 20-45 minutes and was conducted during school hours. The interviews followed a protocol 
determined by the research questions. These interviews provided room for the exploration of the research questions 
with the teachers allowing them to describe in-depth their perceptions and experience with project-based learning. 
The interview protocol was pilot tested prior to data collection. 

All interviews were recorded using computer software called Audacity. The audio files were stored on a 
laptop to be retrieved later for transcription. Each file was transcribed and saved as a Word document. Two of the 
transcriptions were done by the researcher, two were done by a high school student, and one was done by a college 
student. 
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Document collection. Planning materials such as lesson plans and evaluation instruments were also 
collected from some of the teachers. The analysis of these documents followed the document protocol. The collected 
materials varied in the type of information they provided. Nevertheless, they were helpful in corroborating the data 
collected from the interviews. 

Data analysis 

Analysis of the data followed a constant comparative method (Glaser & Strauss, 1967). The inductive 
process of data analysis started by the researcher gathering information through open-ended questions and 
fieldnotes. These were put into themes and categories that became broader through analysis (Creswell, 2003). 

Iterative rounds of data reduction began with open coding directly from the interviews. Similar codes were 
grouped into categories and similar categories were grouped into themes. For example, raw codes were highlighted 
on the text printout of the transcripts then cut out. Each code was referenced by the initials of the participant’s name 
and the line number of its location on the transcript. Following, similar codes were grouped together. Then, these 
codes were transferred to the online visual thinking tool Webspiration that allowed the creation of a concept map. 
Codes were typed in boxes and boxes of similar codes were grouped together to branch into the same category. On 
Webspiration, codes and categories were color coded to facilitate their identification (see Figure 1). Later, similar 
categories were grouped together to form a theme. This process was followed by a peer debriefing where, using a 
whiteboard, themes and subthemes were reviewed and organized. 

The resulting codes and themes allowed for the identification of teachers’ profiles that relate to their use of 
PBL. These profiles were supported by the reflective notes taken by the researcher after each interview, 
highlighting personality traits and first impressions of the teacher’s approach to PBL. 

Lastly, the documents collected from the teachers were studied to look for similarities with the content of 
the interviews on the design of the PBL lessons or activities, evaluation methods, technology integration, or 
examples of artifacts. 

Rigor and trustworthiness 

Four strategies were used to ensure the rigor and trustworthiness of this study. 

Triangulation. In this study, two sources of data collection were used in order to ensure triangulation: 
semi-structured interviews and documents. The semi-structured interviews were based on the interview protocol 
and consisted of open-ended questions. This way, the teachers were able to elaborate with their responses by giving 
their own perceptions and interpretations without much interference from the researcher. The researcher probed and 
redirected the focus of the interview to the research questions. As for the documents, they were studied to look for 
corroboration on the content of the interviews. 

Member checks. Here, the transcripts were sent through an email attachment asking the teachers to review 
them and make changes. Only one teacher responded with no changes. 

Peer debriefing.. Several peer debriefing sessions were held with a professor where codes, categories, and 
themes were discussed as well as the construction of the teacher’s profiles. These discussions were based on concept 
maps and outlines displayed on a white board. Additionally, discussions covered how rigor was achieved. 

Audit trail. In this study, the researcher kept a journal on the coding process and the creation of categories 
and themes. Furthermore, after each interview, the researcher documented her reflections on the personality traits of 
the teachers and their reactions during the interview. These reflections helped in the construction of the teachers’ 
profiles. 



Findings 

The teachers in this study differed in their beliefs on how and when to use PBL. While the sample was 
small, four profiles of PBL teachers emerged from the data. By using PBL, teachers reinforce learning (reinforcer), 
they extend learning (extender), they initiate learning (initiator), or they can navigate among these three trends 
according to need (navigator). 

Table 1 displays the four profiles of the teachers, their names, the type of school they worked in, the grade 
level, and the subject-matter they taught. 
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Table 1 

Profiles and characteristics of teachers 



Profile 


Name of teacher 


Type of school 


Grade level and subject- 
matter 


Reinforcer 


Greg 


Public 


12 th grade English 


Extender 


Audrey 


Public 


9 th grade English 


Extender 


Diane 


Public 


5 th grade Math and Science 


Initiator 


Martha 


Private 


5 th grade Social Studies 


Initiator 


Brenda 


Private 


4 th grade Reading, Math and 
Social Studies 


Navigator 


Scott 


Public 


6 th grade Math and Science 



Reinforcers 

A reinforcer, indicated by Greg, perceives the benefit of PBL as its potential to supplement and reinforce 
content that has been completely taught. 

Greg. Greg is a teacher who enjoys the use of projects in his classes. He strongly believes that it is a good 
approach to motivate students. He uses PBL to increase the students’ interest in the material they are learning which 
they sometimes perceive as dry. In addition, Greg takes advantage of their high technology skills and opens the door 
for his students to make use of them, in another attempt to motivate them. However, Greg uses projects to 
supplement and reinforces his teaching. In this regard he says: 

Well, I think, what I try to do is a supplement to what I’m already teaching. For example, if we’re reading a 
novel or a play, then 1 will test them, of course, over their knowledge of the content. But then, in addition, 
what I try to do is some sort of project-based learning, it sorts of supplements and reinforces the content. 

On using technology he says: 

The way I do that is to do technology and most students do like technology. So, the basic assignment is 
this: they have to find a published poem. . . then they are to determine the theme of that poem and then they 
are to find a song that matches that theme, and then we go into the computer lab and using iMovie and then 

find pictures to match that theme and then marrying all together into an the iMovie So, it just sort of 

supplements to them that they have got to identify the theme and then they’ve got to find the song that also 
matches that. 

This idea of reinforcing and supplementing his teaching with projects is recurrent all through the interview. 
The projects come in after the content has been taught to the students and their knowledge on it has been tested. 

Greg perceives the benefit of PBL in its potential to supplement and reinforce a content that has been completely 
taught. In that sense, Greg reinforces learning through PBL. 

Extenders 

An extender, exemplified by Diane and Audrey, uses PBL by having students problem-solve and use 
critical thinking skills to take their learning a step further from what had been already taught. 

Audrey. Defining PBL, Audrey reveals her constructivist approach to teaching. She says, “Basically it’s an 
opportunity for the students to learn in a different way and to investigate things and work together and put it together 
and just come up with their own solutions.” Audrey chooses to use project-based learning in her class because she 
believes it motivates the students, it engages them in critical thinking and creative work, and provides them with 
authentic learning experiences. However, Audrey sees that project-based learning is best suited to complement the 
content material she is covering in class. She says: 

1 find something that is really interesting to them from what we read about that they have some good 
discussion on. And I just think what would be a good project to create out of that. ... It is complimentary to 
what we are working on already because to me, sometimes it’s difficult to just open a whole new can of 
worms and then, we still have this material to cover. So I go from whatever we are reading or doing in 
class, and then whatever I think can complement that, and help them understand it better. 
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In this respect, Audrey extends learning through PBL. 

Diane. For Diane, PBL is a way for students to leam by doing and produce a product or an artifact that 
shows what they have learned. On the other hand, Diane views the potentials of PBL to be in its use as a culminating 
activity to the units taught in class: 

I have typically put projects towards the end of the unit. I feel like it is more of a culminating activity. . . so 
that they can take everything that they have learned throughout that unit and put it into a project. 

Therefore, Diane does not teach her content through PBL, but her approach is to cover the content first and then 
move to the project next. However, in doing so, her projects are not repetitive of the content but they allow students 
to delve farther, reaching the objectives through critical thinking and learning by doing: 

We wanted to do something where the kids were taking things from different subjects. . .. We knew that we 
wanted them to be thinking critically. We looked at the objectives that were our state standards, what skills 
we wanted the kids to be able to accomplish through this project and then we looked at the materials that 
they have already learned about and what new skills they might need to be taught before they would be able 
to accomplish the project. ... Because they don’t have a lot of exposure to city planning, we did a few 
lessons on what is city planning, what kind of components need to go into that. . . . like a hospital or the 
schools, those types of things, but then we also gave them some choice in terms of optional items. ... We 
told them the components that needed to be presents but how they laid that out was up to them. ... So those 
kinds of things that they discovered were done all on their own so. 

So, basic knowledge and skills were covered before the project. For Diane, this is a necessary step that 
enables the students to use higher order thinking skills and deepen the focus of the content. As a result, students 
would then show their knowledge and skills in the artifact they produce. In this sense, Diane extends learning 
through PBL. 

Initiators 

The profile of an initiator, indicative of Martha and Brenda, launched a unit of learning for their students 
with research questions, continued with a journey of discovery and critical thinking, and led to the production of an 
artifact. 



Martha. Martha explains her use of project-based learning as a method through which learning takes place. 
Talking about a project on Canada, she says: 

[Project-based learning is] the way we cover the unit. We did use our textbook to cover background reading 
on Canada. . .. But I found out that if you do let them search for the information and work with the 
information rather than reading out of the textbook they tend to remember it better and they are learning 
and they are exposed to some many more skills that way. ... 1 just try to introduce them to it and then 
provide them with different places to go to do their own research. But typically, when they find it on their 
own, they learn more from it, and they are more drawn into it. 

Therefore, with PBL, Martha takes her students to where they construct their own knowledge through 
research. In addition, Martha puts an effort in scaffolding their learning, as well as including the element of 
reflection in her PBL activities. On the other hand, a strong element in Martha’s implementation of PBL is her 
multidisciplinary approach towards it. Supported by the whole 5th grade teaching team, Martha is fully aware of 
how a multidisciplinary approach to projects enriches the learning experience of the students and makes it more 
authentic. 

Therefore, Martha is the teacher who uses PBL to guide the learning of her students with research questions 
that takes them on a journey of discovery and critical thinking, leading to the production of an artifact that 
demonstrates their learning. Therefore, she initiates learning through PBL. 

Brenda. The first impression one gets when talking to Brenda is the vivaciousness of her personality. She 
describes her experience with PBL with great excitement and details. In doing so, Brenda reveals a deep 
understanding of the potentials of project-based learning, even without having had any professional development on 
it. This is how she explains how learning happens through her PBL: 

The way I see project-based learning is you need to have a focused idea or set of ideas that you want the 
students to learn but the approach is students learn through different projects. ... It is hands-on learning and 
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learning by doing. That is the way that I would in a nutshell describe it. . .. We [teachers] think, okay, here 
are our goals and then we get into the unit and see that the kids have taken in it in a whole other direction. 
So then, we have to shift with them because that’s their train of thinking. . . they feel successful and proud 
of themselves about what they’ve produced. . . 

Brenda explains how she implemented a PBL activity in covering a unit on Space: 

So my teaching partner and I came up with a set of learning goals that we wanted our students to learn. In 
the beginning, we talked with our science specialist and he mapped out a few specific labs that they can do 
in his classroom to get them started. Then the students did some research in groups about the different 
craters of the moon . . . and then they did something called a VoiceThread on the computer, in partners, 
about a couple other different ideas about the moon. . . they’re currently working on a 3D model of the 
moon. . . . and then after we finish that, we are going to go to The Space and Rocket Center, and they are 
going to do a moon experiment, hands-on experiment. So that’s just one unit we do and we actually do four 
different units throughout the year with project- based learning, so that’s just one example. 

For Brenda, the importance of project-based learning lies in students learning by doing, learning in 
different ways, and showing what they know in different ways, all the while emphasizing strongly a student- 
centered approach. In addition, Brenda sees the potential of PBL in providing the students with authentic learning 
experiences. To her, collaboration and teamwork are of great importance for they prepare her students for the true 
nature of work in the real world. Moreover, Brenda emphasizes the importance of showcasing her students’ work, 
which aligns with her efforts to offer her students authentic learning experiences. She is very resourceful in 
providing showcasing opportunities for the students’ projects. For that purpose, she shares the artifacts with several 
members of the school community, her colleagues, students from other grades, and parents. 

Therefore, Brenda uses PBL to bring authentic learning experiences to her students. The aim of her 
teaching is not only to enrich the student’s knowledge and skills, but she takes it a step further to prepare her 
students, even at a young age, to real life. Brenda also adds the element of reflection to her PBL, and in creative 
ways such as journaling, drawing, and debriefing. Another strong element in Brenda’s use of PBL is her 
multidisciplinary approach and her collaboration with other teachers. On collaborating with other teachers she says, 
“You have to work very closely with your teaching partner, for sure and decide that this is for sure what we want our 
students to learn and what our students need to learn.” 

In conclusion, Brenda is the type of teacher who embraces project-based learning because it aligns with her 
conviction and understanding of what learning is about. With project-based learning, Brenda can bring her students 
to where they learn by doing, use higher order thinking skills, discover, collaborate, reflect, and produce authentic 
artifacts that stem from the different representations of learning among them. She does not use PBL to supplement 
her teaching. On the contrary, she teaches through the use of PBL. In that sense, she initiates learning through PBL. 

Navigator 

A navigator, evidenced by Scot, uses PBL in any of the forms described earlier according to the learning 
needs of the students. 

Scot. Listening to Scot discuss how he integrates project-based learning in his class, one notices his use of 
the term “it depends” almost for every stage of implementation. Whether asked about how he decides to do a PBL 
activity or how he assesses it, or how he forms the PBL groups, his answers are always based on the need for every 
situation. For Scot, project-based learning seems to be a fluid and malleable process. This “it depends” reflects his 
level of comfort in designing project-based learning in his classes. He is continuously assessing where his students 
are and acting accordingly, using projects to fulfill their learning needs. As a result, he sees two potentials for 
project-based learning. One potential is to strengthen weaknesses and insecurities about content matter among 
students. He explains: 

It depends on the children. . ..Like I wouldn’t necessarily have picked up on the phases of the moon till we 
started talking about it, and you get that kind of a glazed-over look, so you just kind of, oh, we need to 
come up with something on this, and that’s what 1 do. 

The other potential is to deepen their level of understanding beyond curricular demands, allowing them to “fly.” He 
tells them, “We’re doing this because you guys are very secure in this, so I’m going to give you a chance to really, 
really fly, really get a chance to show what you know.” In this sense, Scot shows his skills in navigating through the 
curriculum, weaving project-based learning through it, with the ease of an expert. 
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On the other hand, and in order to motivate his students in addition to using PBL, Scot is keen on 
integrating technology in the projects whenever possible. He says, “Well, I tell you what, when it comes to 
technology, they are dialed in, they are amped, they have more available to them and they can move through it 
quicker than you would ever believe.” 

In essence, Scot aims at bringing his students to master their learning by discovery and exploration. For 
him, this becomes possible when students are motivated. He knows that he can achieve both motivation and mastery 
through project-based learning. His eight-year teaching experience enables him to navigate and place his projects 
strategically in the curriculum where needed. Therefore, Scot is the “navigator.” 

Summary 

The profiles of the participants reveal that those teachers who were selected for this study for their 
exemplary work in project-based learning share common characteristics in some aspects of it but differ in others. 
They all aim at making the learning experience of their students more interesting and more meaningful, and they all 
involve their students in hands-on activities. However, they differ on their perceptions of how PBL should support 
learning. 

These participants reveal that PBL can be used at different points in the process of learning. It can be used 
to initiate the construction of learning from minimum background knowledge, it can be used to extend learning after 
core elements have been taught, it can reinforce learning and strengthen it after the content has been taught, or it can 
be used in any of these three forms to meet the needs of the students. 

Implications 

In the absence of professional development in PBL, the teachers in this study practice PBL based on their 
perceptions and beliefs on how optimal learning can be achieved. It is apparent that they appreciate the 
constructivist characteristics of PBL. They use it in their classes because of the advantages it brings to the leaning 
process when compared to the traditional didactic approach. They want their students to use higher order thinking 
skills, they initiate social learning, they ask their students to show their knowledge through the production of 
authentic artifacts, and they assess the outcome of learning outside the limitations of traditional testing. This is 
precisely what the literature on constructivism implies in that it aims at the construction of knowledge with multiple 
perspectives and within a social activity. It is context dependent, and it allows for self-awareness of learning and 
knowing (Duffy & Cunningham, 2005). 

However, the differences observed in how teachers implement PBL, whether reinforcers, extenders, 
initiators, or navigators, reflect their teaching and learning philosophy which is shaped by their beliefs about the 
effective use of PBL. Ertmer (2005) points out the confusion around labeling and defining the beliefs of teachers. 
She states that this confusion is due to the difficulty in determining the difference between pedagogical beliefs and 
knowledge. She also states that beliefs cany an affective element absent in knowledge. These differentiations are of 
particular importance in this study in explaining why teachers differ in how they use PBL. 

At the affective level, all the teachers embrace PBL as a teaching model. Therefore, they carry positive 
pedagogical beliefs about it. At the knowledge level, these teachers understand constructivism, but they are not as 
equally knowledgeable about the systematic implementation of PBL. They implement PBL to the best of their 
abilities without any professional development in its particularities. Therefore, the difference observed in how they 
implement PBL may be due to the lack of an in-depth exposure to what it can bring to the learning process. This 
difference may also be due to a strong belief about where PBL can best be placed on the continuum of the learning 
process. 

Moreover, teachers’ use of PBL may reflect their comfort level in creating a balance between curriculum 
and testing needs on one hand and their aspirations towards employing constructivist strategies on the other. To that 
regard, Ertmer (2005) notes the importance of sorting out how teachers’ beliefs affect their practice. Whether 
reinforcing, extending, initiating learning, or navigating through all of three uses, the teachers’ perceptions are that 
the manner with which they use PBL is proving to be beneficial and successful in their respective classes. 

An important question that poses itself here is whether all teachers should be encouraged to become 
initiators or if PBL could in fact be implemented effectively in any of these uses? Moreover, can PBL use be 
regarded on a continuum starting from reinforcement of learning ending in initiation of learning? Thomas (2000) 
states that for a project to be considered a project-based learning activity, it should be central and not peripheral to 
the curriculum, where students struggle with the concepts of a discipline, and where they construct and transform 
new skills and understandings. However, do Thomas’ recommendations contradict the concept of PBL’s use over a 
continuum? Comparing learning achievement of students between the different uses of PBL emerging from this 
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study as well as providing professional training for teachers in PBL followed by tracing changes in their use may 
shed the light on these questions. Ravitz (2010) posits, “no two teachers implement PBL the same way” (para. 10). 
He also states, “it does not seem reasonable to expect teachers to learn about and use this approach entirely on their 
own. . . . Effective use of PBL requires extensive planning and professional development” (para. 12). 

Professional development offers teachers with the strategies, confidence and guidance that they need in 
order to incorporate PBL effectively in their classrooms (Ravitz et al., 2004). In addition, professional development 
can provide teachers with the skills needed to overcome some of the banders they face, such as time limitations 
(Hertzog, 2007) and striking a balance between curriculum requirements, testing policies and PBL (Krajcik et ah, 
1994; Snyder & Snyder, 2008). Moreover, professional development may also lead to a shift in the teachers’ beliefs 
on how PBL can be best placed on the learning continuum. Finally, professional development may sort out the 
reasons behind the different uses of PBL observed in this study, whether it is knowledge-based or belief-based. 

Limitations 

This study has several limitations. First, because of the purposeful sampling and the small sample size, its 
findings cannot be generalized to all teachers unlike a study in which the sample has been randomly selected. 
Second, for some participants, the duration of the interviews was short. Since these interviews were done on site and 
during school hours, the length of the interview was limited to the time available for teachers in between classes. A 
longer interview would have allowed for more probing and would have generated more data. Third, only one 
interview per teacher was carried out. Subsequent interviews would have permitted follow-up on the preliminary 
findings with the teachers and would have enriched the data further. Fourth, the data was collected through 
interviews and document collection. However, in-class observation of how PBL is being implemented would have 
also added another perspective on the findings. The limitations in the data collection process were due to the data 
being collected towards the end of the school year that would have made scheduling subsequent visits difficult. 
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Abstract 

A framework is proposed that matches two popular forms of social media with the types of learning they afford. 
Kaplan and Haenlein’s (2010) taxonomy is expanded to consider how learners participate in social media (based on 
Ito, et al., 2010) and how participation styles connect to general and discipline-specific learning. This framework 
can help researchers ask more precise questions about learning with social media, and can also provide guidance for 
teachers wanting to use social media with their students. 

The term social media is familiar to most people, and there is a great deal of interest in how these popular 
forms of new media can facilitate learning. The term is broad, however, and encompasses many different types of 
applications with different affordances. Instead of asking "how can we use social media for learning?" researchers 
need to ask more precise questions, such as “what types of social media are the most helpful for what kinds of 
learners and what kinds of learning?” This precision can be difficult to achieve, however, in an emerging field where 
the terminology is still evolving. Kaplan and Haenlein (2010) provided a useful classification of social media sites 
based on the structure and features of each medium. In this framework they made a clear distinction between social 
networking sites (e.g., Facebook and MySpace) and content communities (e.g., YouTube and Flickr). While their 
framework was designed for the purpose of business and marketing and does not address how the features of these 
sites relate to learning, it does provide a useful starting point for analyzing the features and affordances of these two 
similar but distinct forms of social media. 

Education researchers have begun to explore the potential of social networking sites for learning 
(Greenhow & Robelia, 2009) and also how to use the video content available on content communities such as 
YouTube (Burke & Snyder, 2008; Niess & Walker, 2009; Webb, 2010). Only a few studies (Birch & Weitkamp, 
2010; Thompson, 2010) have emphasized both media and integrated comments on social media sites as a way to 
support learning in traditional school subjects. Ito et al. (2010) looked at social media from a learning perspective 
but focused on how people participate in social media rather than on the features of the sites. Their analysis adds 
greatly to our understanding of how people participate in social media, but offers only indirect guidance on 
matching a specific media type to a desired learning goal. 

This paper expands on the definition of a content community (Kaplan & Haenlein, 2010) by incorporating 
the work of Ito et al. (2010) and research on learning in the disciplines (Langer, 2011; Langer, Confer, & Sawyer, 
1994). It presents an analysis of how social networking sites and content communities afford different types of 
participation and interaction, and how these different affordances may support different types of learning. 
Specifically, it proposes that while social networking sites primarily provide opportunities for learning social skills 
and developing identity, content communities have affordances that can facilitate discipline-specific learning. 

A Classification System for Social Media 

Kaplan and Haenlein (2010) classify social media types into six categories based on a three-level (low, 
medium, and high) social presence/media richness scale and a two-level (low and high) self-presentation/self- 
disclosure scale. According to this scale, applications such as blogs and wikis feature a low level of social presence 
and media richness, while virtual social worlds and virtual game worlds provide very high levels of social presence 
and media richness. Applications such as Facebook, MySpace, and YouTube, which are among the most popular 
media types with teenagers and young adults today (Nielson, 2009), are identified by Kaplan and Haenlein (2010) as 
having a medium level of social presence and media richness. Within this last group of applications, those requiring 
a high level of self-presentation and self-disclosure (e.g., Facebook) are labeled as social networking sites and those 
requiring less self-presentation and self-disclosure (e.g., YouTube) are called content communities. 
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The framework provided by Kaplan and Haenlein (2010) is a useful because it provides a precise way to 
label sites like YouTube and Flickr as belonging to a distinct category, separate from social networking sites like 
Facebook and MySpace. When classifying social media for learning, however, we need to look beyond the design 
features of the site and examine the type of content featured, how users interact with the content, and how users 
communicate with each other about the content. 

Attributes of Social Networking Sites and Content Communities 

On the surface social networking sites and content communities may appear to be very similar. Both types 
of sites allow users to create profiles, create connections to other users (e.g., "friend" or "follower" relationships), 
and comment on each others' contributions to the site. In fact, some early sites that began as content sharing sites 
gradually added social features and began to look more like social networking sites (boyd & Ellison, 2008). At the 
same time, the difference in the levels of self-presentation and self-disclosure noted by Kaplan and Flainlein (2010) 
present important differences between these two types of online communities. In a social networking site self- 
presentation and self-disclosure constitute the primary content on the site, whereas in content communities the 
content being shared (e.g., videos, photos, etc.) is the primary content. This shared content then provides the space 
where conversation between members occurs. Whereas in a social networking site conversation occurs in the 
comparatively private space of an individual member's "wall" or profde page, in a content community the interaction 
between members occurs in a public space within the site. Conversations occurring in public spaces are more likely 
include contributors outside each member's usual social circle, and to include contributors of different ages and with 
varying levels of expertise. These different attributes of social networking sites and content communities afford 
different kinds of communication and thus different kinds of learning. 

Types of Social Media Participation 

Ito, et al. (2010) defined two categories of participation in social media: friendship-driven and interest- 
driven, based on the users' motivation for engaging with a particular site. Friendship-driven participation refers to 
"the dominant and mainstream practices of youth as they go about their day-to-day negotiations with friends and 
peers" (Ito, et al., 2010, pp. 15-16) and is thus mainly social interaction between people of the same age. Social 
networking sites, where social interaction occurs on participants' personal space (e.g., the Facebook wall) and self- 
expression is the main content, provide an environment well suited to this type of participation. Ito et al. state that 
social network sites "are supporting those sometimes painful but important lessons in growing up, giving kids an 
environment to explore romance, friendship, and status just as their predecessors did" (Ito, et al., 2010, p. 22). 

Online social networks provide a place for teens to gather and "use specific media as tokens of identity, taste, and 
style to understand and display who they are in relation to their peers" (Horst, Herr-Stephenson, & Robinson, 2010, 
p. 41). The communication and media sharing capabilities of social networking sites facilitate this type of 
interaction. 

Interest-driven participation centers on a common interest such as online gaming, sports, or music. In 
interest-driven forms of participation, people "develop deep friendships through. . .interest-driven engagements, 
but. . .the interests come first, and they structure the peer network and friendships, rather than vice versa" (Ito, et al., 
2010, p. 16). Content communities such as YouTube, Goodreads, and TED are structured in a way that can facilitate 
interest-driven participation. On these sites interaction occurs in response to the content presented (e.g., the video or 
book being discussed) and opinions or insights about the content are the focus of the discussion. In addition, they 
can provide access to a community of expertise in the subject of interest, which is important for supporting high 
levels of interest-driven participation (Horst, et al., 2010, pp. 74-75). 

Based on the distinctive affordances of these two types of social media sites, I propose that while social 
networking sites lend themselves well to developing communication skills, digital literacy, and identity (Greenhow 
& Robelia, 2009; Ito, et al., 2010), the interest-driven style of conversation that can occur in content communities 
affords an opportunity to learn discipline-specific ways of thinking and reasoning (Langer, et al., 1994), making 
them a promising tool for supporting engaged learning in traditional school subjects. 

Content Communities and Disciplinary Learning 

Research on learning in the disciplines suggests that, in addition to general skills such as critical thinking 
and problem solving, there are discipline-specific ways of thinking and reasoning that students need to acquire. 
According to Langer, "familiarity with language, structure, and disciplinary conventions are at the root of learning in 
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academic coursework" (2011, p. 4). Langer and her colleagues (Langer, 2011; Langer, et al., 1994) have written 
extensively on the differences in teaching and learning among various academic disciplines, but for illustrative 
purposes in this paper I focus on how content communities may support two aspects of disciplinary learning: (1) 
learning the vocabulary of the discipline, and (2) learning discipline-specific ways of making claims and selecting 
evidence. 

Discipline-Specific Vocabulary 

Content communities provide a way for learners to be exposed to and practice using the language of a 
discipline, since even the most informal posts can contain technical vocabulary. For example, the conversation 
below comes from comments attached to a YouTube video featuring guitarist Gary Floey. The words in bold are 
technical terms related to guitar technique and references to other guitarists (Eddie Van Flalen and Joe Satriani) that 
a serious guitarist would be expected to know. 

"dude seriously though., how did you get to sound so much like eddie [.sic]??? that's like my ultimate goal. 
(WammyHammy, 2008) 

"he also uses an auto wah pre set in here hence that Satrianiesque sound as well." (carsales34, 2008) 

"half of EVH's sound is that flange/phase effect, and other than that it's technique and I don't mean that to 
belittle Eddie at all, he's absolutely amazing!" (zephanizzleiah, 2009) 

This informal conversation would allow a novice guitarist to experience the vocabulary and style of 
speaking about music common among other "guitar geeks," and thus begin to learn the unique terminology of the 
craft. 

The disciplines taught in school also have their unique vocabularies that students need to learn, and content 
communities can provide opportunities for exposure to these vocabularies. For example, in the following 
conversation attached to Dr. Bonnie Bassler's TED video on bacterial quorum sensing, the following post uses the 
vocabulary unique to biology: 

If you really want to know more about how bioluminescent bacteria communicate, you should look up the 
lux operon on Google. The lux operon is a set of genes within the DNA of Vibrio fischeri that regulate 
the expression of the chemical (homoserine lactones) signals that the bacteria use in quorum sensing. 
This is just one example of how quorum sensing is regulated, however, there exists a plethora of examples 
within other species of bacteria that regulate quorum sensing in a very similar manner. (Derek Boyer, 

2009, April 10) 

Conversations such as these can expose students to people outside the school environment engaged in conversation 
that is partly social in nature but also models the use of discipline-specific vocabulary. 

Discipline-Specific Ways of Using Evidence 

Langer, et al. (1994) discuss several aspects of what it means to "think in the discipline" but one important 
aspect is the type of evidence the discipline views as appropriate and convincing, and the closely related concept of 
what type of evidence teachers look for in assessing their students learning. For example, in biology, they found that 
teachers emphasized a proper understanding of parts of a system and how the parts functioned within the system. 
This type of systems-related thinking is exemplified in the TED comments linked to the bacterial quorum sensing 
video as the commenters debated the promise and pitfalls of anti-quorum sensing dings to cure disease: 

she didn't really explain the mechanism of the treatment, but the simplest would be that flooding the 
bacteria with these signals would cause them to act before they had the numbers necessary to cause 
damage, and then the immune system could fight them, the other possibility would be that these signals 
would bind on the sites and interfere with them from acting at all. (jason_brann, 2009, April 8) 

In American history, in contrast, Langer, et al. (1994) found that teachers expected students to "provide the 
facts, as well as the students' explanations and interpretations - through similarities and contrasts, by explaining 
connections across time, cultures, and situations" (p. 32). In a discussion on the TED site in response to James 
Watson's talk on the discovery of DNA, participants argue about the role Rosalind Franklin played in the discovery. 
In the course of their discussion they draw attention to the appropriate interpretation of facts relative to their 
historical context. 
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Franklin is not the victim of some conspiracy by less talented people, she has become an icon for the ailing 
feminist cause who are now struggling to find reasonable examples of overbearing male oppressors 
depriving their sisters of historical importance. ...yawn! Franklin did not do the bulk of the work and lacked 
the essential insights that Watson and Crick provided, (j michaels, 2010, April 11) 

Your history and logic are awfully weak. Flave you read the book that A. Brown recommends above? Do 
you really think the contributions by women were taken with the same seriousness as those by men in 
Cambridge in those days? Flow does being held up as an example by feminists affect the fact that Franklin 
received less credit for her work than she deserved? No logic there... In my field of astronomy women were 
not even allowed to use the world's largest telescopes until the late 1960s... (DouglasDuncan, 2011, May 
14) 

These two examples demonstrate conversations from the same website (TED.com) but in two different 
disciplines, and reflect some of the discipline-specific differences in gathering and presenting evidence identified by 
Langer and her colleagues (1994) in their study of teaching and learning in the disciplines. 

Conclusion 

The examples shown here were chosen purposefully to illustrate the discipline-specific differences and the potential 
of content communities to support some aspects of discipline-specific learning. This does not mean that every 
discussion thread or every content community has the same quality of discussion. The educational potential of 
comments on the TED website, for example, may be particularly high and not typical of other content communities 
such as YouTube (Thompson, 2011). Nevertheless, because content communities, especially those featuring video, 
are popular with the current generation of students (Nielson, 2009), teachers and researchers should not overlook 
content communities as a unique type of social media, distinct from the better known social networking sites, with 
affordances that can be harnessed to support discipline-specific teaching and learning. 
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Blended Learning as the Future of Teacher Education: 

A Preliminary Study of Faculty Belief 

Ying Wang 

Abstract 

This preliminary qualitative study aims to examine if faculty members with and without online teaching experiences 
differ in how they perceive online learning in teacher education. Eight teacher educators from two institutions, five 
of whom have never taught online, participated in semi-structured interviews. Findings indicate that faculty 
members with and without online teaching experience share similar concerns about online teacher education 
programs and believe blended learning is the future of teacher education. 

Introduction 

The U.S. National Education Technology Plan (U.S. Department of Education, 2010) emphasizes using online and 
blended learning to empower learning at all school levels. Researchers and practitioners are examining reasons for 
the slow progression of online learning into mainstream academy (Puzziferro & Shelton, 2009). Faculty acceptance 
is regarded as a determining factor for the success and failure of online learning in higher education (Major, 2010). 
Major’s study indicates that university faculty with experiences in online distance education believe that teaching 
online changes the way they approach and think about teaching, course design, time, instruction, and students. 

Findings from studies on online learning in teacher education show that there is no significant difference between 
online and F2F instruction (Caywood & Duckett, 2003; Jordan, et. al., 2004; Steinweg, Davis, & Thomson, 2005; 
Skylar, et. al., 2005); students generally respond positively toward online learning for its flexible time scheduling, 
freedom in expressing ideas and comments online, elimination of non-essential conversation in online discussion, 
and increased confidence using technology in instruction (Beard & Harper, 2002; Hughes & Hagie, 2005); the 
instructor plays an important role in supervising and encouraging student interaction and collaboration in online 
courses; and blended learning holds potential for developing interactive and collaborative learning communities for 
teacher candidates (King, 2002). Blended learning combines online and face-to-face (F2F) learning to achieve the 
inherent benefits of both types of instruction (Osguthorpe & Graham, 2003). Such an approach is also labeled as 
hybrid learning. 

Research about online and blended learning is still lacking in teacher education, especially on the application of 
online and blended learning at the program level. The purposes of this preliminary study are to generate discussion 
and identify further research directions for applying online and blended learning at the program level in teacher 
education. Faculty belief is chosen as the focus for this preliminary study. The research questions include: 

(1) Do faculty members with and without online learning experiences perceive online learning in teacher 
education differently? 

(2) How do faculty members envision the future of online learning in teacher education? 

Participants 

Eight faculty members from two institutions’ teacher education programs, five from a private college and three from 
a state university, participated in semi-structured interviews. Both institutions are located in the Midwestern United 
States. All of the participating faculty members have been teaching for more than five years in their area. The 
private college has traditional residency undergraduate teacher education programs with approximately 350 students 
enrolled in its nine teaching licensure programs. The state university has traditional residency undergraduate teacher 
education programs, online graduate programs, and a blended learning program where teacher candidates meet on 
campus for two days at the beginning and end of each semester with the remaining class time convened online. 
Currently 460 students are enrolled in the university’s undergraduate teacher education programs, including 220 in 
the residency program and 240 in the blended program, and 150 in graduate programs. 
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Data Collection and Analysis 

This is a preliminary qualitative study. Five interviews were conducted in person and three via telephone. Each 
interview lasted 30 to 45 minutes. Interview questions were focused on participants’ perception of online learning in 
teacher education and their vision for teacher education in the digital age. Interview data were analyzed qualitatively 
to draw common themes. 

Results 

Five of the participating faculty members had no experiences with online teaching and three of them had 
experiences teaching face-to-face, online and blended courses (see Table 1). Through comparing faculty responses 
regarding the benefits, challenges, and future of online learning in teacher education, common themes emerged on 
their concerns about online teacher education programs and their beliefs related to blended learning as the future of 
teacher education: 

Table 1 



Participating Faculty Information 



Faculty 


Gender 


Online Teaching Experiences 


A 


Male 


No online teaching experience 


B 


Female 


No online teaching experience 


C 


Female 


No online teaching experience 


D 


Male 


No online teaching experience 


E 


Female 


No online teaching experience 


F 


Female 


Online teaching experiences 


G 


Male 


Online and blended teaching experience 


H 


Female 


Online and blended teaching experience 



(1) Faculty members with and without online teaching experiences share similar concerns about online teacher 
education programs. A major concern is the lack of modeling and hands-on experiences of classroom instruction 
when preparing teachers online: 

“Teaching is a people business. There is an art and skill to it that is modeled in how we teach our classes. 
The one class I mentioned that is co-taught. It co-taught on purpose, because we realized we didn’t have a 
class where we modeling co-teaching, and our students will often be, once they are out, in a co-teaching 
situation. And we model that in how we teach that course. . . I just don’t think that online I can have my 
students experiencing all the sensory things I bring in and, you know, the different, wide assortment of 
books I bring in for them to look over and evaluate what makes a good children’s book. That kind of stuff - 
I think the quality of that would suffer” (Faculty B). 

“Teaching is a social, F2F activity, and we have to remember that. . . a lot of the younger generation can 
only communicate by texting or on a cell phone or on a computer and not actually talk to each other F2F. 

So you might have a teacher candidate that because everything was online, can look good. But you put that 
person in an actual room with twenty to thirty bodies, it’s a whole other issue” (Faculty F). 

“How do you model the methods? . . . Many of them haven’t experienced good cooperative learning. They 
can’t just read about it or watch a video. They have to experience learning that way themselves to get it. I 
think that with the methods I do, much of the methods is caught rather than taught. If I teach classes in the 
way I want them to teach middle school, high school kids in the classroom, they catch it” (Faculty C). 

Related to faculty’s belief of teaching as interpersonal profession, the lack of community building in fully online 
teacher education programs concerns them: 

“You can’t build the community that you need. You can’t do the hand-holding and the cheerleading that’s 
necessary” (Faculty H). 

“I think a disadvantage would be [missing] the F2F interaction and access of community. Um, a transaction 
would be lost., if everything were online” (Faculty E). 
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Communication issues such as misinterpretation due to the lack of F2F interaction are recognized and shared among 
teacher education faculty regardless: 

“It’s communication. I think to me that’s the number one thing. I think content can come across online, but 
I think a lot of times questions come up, and I know the chatting thing can happen too, but I really feel that 
even with emailing, or chatting, or anything done online, there’s always this small amount of space that you 
have to leave for interpretation and sometimes interpretation can be different depending on how you read 
something” (Faculty A). 

“Sometimes when I do something online and I’m using precise English, formal English, I still don’t get my 
message across. They misinterpret it. And so I have to do a webinar to make sure I get everything cleared 
up. And when you are doing face-to-face they can tell by your body language what you are saying” 

(Faculty F). 

(2) Faculty members with and without online teaching experiences share the belief that blended learning with more 
field experiences to bridge theory and practices is the future of teacher education. Participating faculty were asked 
about their ideal teacher education program. Seven of them expressed explicitly that blended learning is the direction 
to go: 



“I think they’ll - the residency programs will go more hybrid. I think they’ve got an opportunity, since this 
exists already for younger students and older students” (Faculty H). 

“It would be blended with some things online and some things F2F, and I think that it would allow, it 
would be a program that would allow for international teaching to be happing” (Faculty E). 

“I think there’s also a place for blended learning. . . I don’t think we’ve quite arrived there yet as to how that 
will stabilize, so I think that’s still on the rise and I think that it will continue to be on the rise. I think it 
benefits, probably benefits everybody. There’s a place for doing work independently and online, and 
there’s a place for coming together in certain courses” (Faculty A) 

“I can see parts of it, perhaps, being online, but the whole thing. . . not so much” (Faculty B). 

“I would probably do a hybrid one. I would probably do at least six meetings in a regular semester where 
all the students met together. . . And then probably something I would emphasize that each student would 
have at least one opportunity during the semester to have a face-to-face meeting with me” (Faculty G). 

“A blended approach would ultimately be more effective than a purely online approach. It’s just because I 
can’t see how not to have that F2F piece” (Faculty C). 

“We will probably end up into a more of a blended type of situation, where well take the best qualities of 
that online program and instill that where we can, but also realizing that there are certain qualities of the 
education program that still are best done by a F2F kind of meeting time and those aspects will be still 
implemented that way because we realize that that’s where it needs to be” (Faculty D). 

One faculty member who was currently teaching online courses advocated for all teacher education programs to be 
F2F: 



“Yes, [it should go back to the F2F setting, a residential model for all teacher education programs]. . . they 
need to have good material foundation before they deal with the technology, because if you don’t know 
what you are doing and you’re more concerned about how you’re doing it, how well were you going to do 
what you actually have to do?” (Faculty F). 
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Faculty members also emphasize the importance of more field-based experiences for teacher education programs: 

“I would have a lot of field experiences, where teacher education students are actually in classrooms with 
real children, real teachers” (Faculty B). 

“Lots of blending of a practical, field-based stuff with, you know, with classroom, I mean, with instruction 
from an experienced professional. . . more that connect directly to the practice” (Faculty C). 

“They’re talking about - and 1 think it’s brilliant - an intern program, where you’ve done all your generals, 
and then you commit to do something that’s intensively in the field. So you’re in the filed three times a 
week for four or five hours. And then do your education courses as you go along, which 1 think is a brilliant 
idea, because I don’t think they get enough classroom time” (Faculty FI). 

Conclusion and Implication 

Teacher educators who have online teaching experiences may not differ much in their perceptions of online learning 
in teacher education from those who do not have teaching experiences in online environments. Previous research 
findings discussed challenges of online learning in teacher education, such as the lack of direct student-instructor 
and student-student interaction, possible content diminishment due to the emphasis on technology, and difficulties 
caused by students’ lack of technological skills (Beard & Harper, 2002; Flughes & Flagies, 2005; Steinweg et al., 
2005, Stephen & Barford, 2005). Findings from this study support and expand previously identified challenges 
related to online teacher education, such as the potential for misinterpretation caused by the lack of interpersonal 
immediate communication; lack of community building; and lack of modeling and hands-on experiences in 
classroom instruction. Interviewed faculty members with and without online teaching experiences share these 
concerns and believe they may affect the quality of teacher education programs. 

Benton (2008) calls for faculty to go beyond arguing for or against online learning and to experiment the 
possibilities of new technologies. He calls for a variety of instructional approaches to blending and blurring the 
boundaries between old and new media, online and face-to-face contexts, as well as teacher and student roles for 
learning. One of the advantages of blended learning is that it optimizes the advantages of both online and classroom 
instruction (Sitzmann et al., 2006). Participants in this study highlight the potential benefit of applying blended 
learning in teacher education. Both faculty members who are currently teaching online and those who have never 
taught online believe blended learning is the direction of future teacher education programs. 

Traditional teacher education programs have been criticized for low quality, incoherent programs, unbalanced 
curriculum reflecting the gap between theory and practice, and poor student field placements (Levine, 2006; Russell 
& McPherson, 2001; Whitney, Golez, Nagel, & Nieto, 2002). Teacher educators in this study emphasize and 
advocate more field experiences for teacher education programs, regardless of F2F, online, or blended format. To 
create high-quality blended teacher education programs, teacher educators face the challenges of balancing online 
and F2F components in the program and providing adequate field-based experiences for teacher candidates. Further 
research studies are needed to address such challenges. 

Limitations & Future Directions 

This study is a preliminary study with limited participants. Faculty members participated in this study on a voluntary 
basis and do not represent all faculty members in teacher education. Faculty members in teacher education possess a 
variety of opinions regarding the benefits, challenges, and future of online learning in teacher education. Additional 
research is needed to include a large sample of teacher educators to further study their perceptions and acceptance of 
online learning. In addition, the research team plans to examine student perception towards online and blended 
learning in teacher education. Results of this study will be shared with participating faculty. Their feedback and 
comments will also be considered for future research directions. 
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Abstract 

This paper critically examines the current state of research on the use of instructional design and 
development (IDD) procedures to develop work-related training that is inclusive of all age groups. The approach 
included a Best Evidence Synthesis of books, academic journals, and conference proceedings in the fields of 
instructional design and technology, training development, and human resource management published since 2000. 
The review reveals a focus on age-related generalizations about cognition and learning capacity as the basis for 
design decisions. However, the role of training context and content is unclear, limiting generalizations about design 
and training effectiveness. Implications for professionals charged with developing and implementing effective 
training strategies for an increasingly multigenerational workforce are discussed, along with review limitations and 
opportunities for further research. 



Introduction 

Despite challenging economic times, organizations continue to invest in employee training and 
development. Studies of industries in the U.S. and in other countries have shown that investment in training is linked 
to increased employee productivity and to organizational profitability (Huselid, 1995; Lyau & Pucel, 2008; 
Sepulveda, 2009). From an employee perspective, the increasing pressures of technological change and the need for 
continuous training to remain competitive in the 21 st Century job market have become key drivers for a workforce 
that is increasingly becoming multigenerational (Rothwell, Sterns, Spokus, & Reaser, 2008). However, opportunities 
for and participation in work-related training is not yet multigenerational, posing challenges for professionals 
charged with developing and implementing effective training strategies. 

In exploring the training gap between younger and older workers, three themes are prominent in the 
training literature: Age-related stereotypes, older workers’ training self-efficacy, and training motivation. Further, 
the literature appears to indicate an interrelationship among these three themes. Posthuma and Campion’s (2009) 
review of 1 17 research articles and books that deal with age stereotypes in the workplace identifies resistance to 
change, lower ability to learn, and lower return on training investment due to short tenure as being among the many 
stereotypes. In a review of 93 articles in adult education journals from 1980-2006, Chen, Sim, Moon, and Merriam 
(2008) found that older adults have been portrayed as a homogeneous group in terms of age, gender, race, class, 
ethnicity, and able-bodiedness, contributing to the growth of age-related stereotypes. The impact of age-related 
stereotypes on training and development opportunities has been explored at the individual country level, as well as 
in cross-country comparisons (Chiu, Chan, Snape, & Redman, 2001; Armstrong-Stassen & Templer, 2005; Harper, 
Khan, Saxena, & Leeson, 2006; Kluge & Krings, 2008; Fouarge & Schils, 2009). Studies that explore cognitive and 
physical declines as part of the normal aging process (Avolio & Waldman, 1994; Verhaeghen & Salthouse, 1997; 
Masunaga & Horn, 2001; Thornton & Dumke, 2005) appear to affirm some of these beliefs about which age groups 
are more or less “trainable” than others. 

In terms of older workers’ self perceptions, there is some consensus in the literature that training self- 
efficacy is closely related to organizational experience and the presence/absence of age-related stereotypes in the 
workplace. For example, the results of a survey of 715 managerial and supervisory employees of a state employer 
located in the Midwestern U.S. indicate that late career stage employees (50 years of age or older) possess lower 
levels of self-efficacy beliefs with respect to success in training, and lower perceived training utility than those in 
early career stages (Guthrie & Schwoerer, 1996). Enhancing older worker training self-efficacy requires re-tooling 
work-related training to enhance feelings of control, minimize dependence on memory, emphasize the application 
and usefulness of the training, and provide an encouraging, non-threatening environment with constructive feedback 
and support (Guthrie & Schwoerer, 1996; Yeats, Folts, & Knapp, 2000; Charness, Czaja, & Sharit, 2007; Ng & 
Feldman, 2008). Age-related stereotypes are also influences on older worker motivation to participate in training. In 
a meta-analysis of 106 peer-reviewed journal articles published between 1989 and 1999, Colquitt, LePine, and Noe 
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(2000) found that age was a key factor in the motivation to learn, with older learners demonstrating lower 
motivation, lower learning, and lower post-training self-efficacy. Hardre (2003) notes a lack of focus on motivation 
in examining training for all age groups, not just older workers. From a legal perspective, age-related effects on 
training and development opportunities could become increasingly actionable, particularly in the U.S., where 
litigation often complicates labor relations (Maurer & Rafuse, 2001). Aside from the potential for legal challenges, 
the removal of age-related barriers to work-related training opportunities is deemed essential to employee 
motivation and retention (Hirsch, 2007), and to a reduction in costs to the organization for absenteeism and work 
injuries (Brooke, 2003). 

Despite worldwide recognition of the changing demographics of the workplace, and the need to keep all 
workers current in the skills/competencies and technologies required to continue contributing to the changing nature 
of work, the extent to which training is designed to be multigenerational and thus, inclusive of older workers, is 
unclear. This paper critically examines the main ideas, models, and studies of the use of systematic instructional 
design and supporting technologies to develop work-related multigenerational training. 

Method 

To analyze the current state of training design inclusive of older workers, the search strategy used for this 
paper is an adaptation of the Best Evidence Synthesis (BES) approach (Slavin, 1986). This search strategy enables 
the researcher to work with whatever evidence is available, taking account of study design and quality, and use a 
range of approaches to synthesize the heterogeneous evidence. The best evidence strategy involves several 
iterations, because the initial, inclusive search generally produces a very large number of citations that cannot easily 
be excluded automatically by means of search filters. Beginning with wide inclusion criteria may still result in 
locating few evaluative studies of any design, so that the final number of studies to be reviewed in depth is in the 
tens, or fewer, depending on the extent to which the topic area has been a focus of research (Petticrew & Roberts, 
2006). 

Using keyword combinations such as “older workers”, “older employees”, “training”, “multigenerational 
workforce”, and “designing instruction”, a search was conducted of the Psychlnfo, ISI Web of Science, JSTOR, 
Psychological and Behavioral Sciences and ProQuest electronic databases for peer-reviewed English-language 
journal articles published from January 2000 to April 2010, yielding 609 citations. A search of the “gray” literature 
(e.g., dissertation abstracts, conference proceedings, trade magazines, unpublished reports) (Petticrew & Roberts, 
2006) yielded another 143 citations, for a total of 752 citations. 

After a merge-purge of duplicates, book reviews, and editorials, abstracts of 537 studies were retrieved and 
read for relevance. A reading of the abstracts revealed 456 studies that did not address age, training, or focused on 
geriatric care training. Another 64 were excluded because the results presented did not relate to designing training 
inclusive of or targeted to older workers. In total, 17 studies qualified for inclusion in the analysis. The 17 studies 
(see Table 1) that were reviewed and analyzed address two broad issues: (a) models for designing instruction 
inclusive of older workers (7 studies) and, (b) training performance outcomes with instructional design models 
specifically targeting older workers (10 studies). 

Table 1. Multigenerational, age-inclusive training 



Themes 


Citations 


Models for designing age-inclusive training 


Pate, Du, and Harvard (2004) 

Ford and Orel (2005) 

Koc-Menard (2009) 

Newell, Arnott, Carmichael, and Morgan (2007) 
Rothwell, Sterns, Spokus, and Reaser (2008) 
Fisk, Rogers, Charness, Czaja, and Sharit (2009) 
Van Gerven, Paas, and Tabbers (2006) 



(Table continues) 
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Table 1 (Cont’d) 



Themes 


Citations 


Training performance outcomes with design models 
targeting older workers 


Callahan, Kiker and Cross (2003) 

Chilarege, Nordstron, and Williams (2003) 

Sharit, Czaja, Flernandez, et al. (2004) 

Stolze-Loike, Morrell, and Loike (2005) 

Wallen and Mulloy (2006) 

Themistocleous, Koumaditis, Mantazana, Morabito (2010) 
Jamieson and Rogers (2000) 

Sanders, Gonzalez, Murphy, Pesta, and Bucur (2002) 

Nair, Czaja, and Sharit (2007) 

Ownby, Czaja, Loewenstein, and Rubert (2008) 



Findings 



Models for Designing Age-inclusive Training 

Critical to the design process is a solid understanding of the characteristics of the learner, particularly the 
learner’s ability to process and apply information to effect a change toward improved performance. To that end, one 
stream of thought advocates targeting the specific needs and limitations of older workers as a means of 
multigenerational training design. Drawing on the fields of cognitive psychology and instructional design, Pate, Du, 
and Havard (2004) state that instructional designers must become more aware of ageing and the cognitive learning 
needs of older adults because of demographic changes in the workforce. Beyond current skills, abilities, and 
technical competence of older workers, designers need to understand barriers to learning such as health factors, 
cognitive changes affecting memory and concentration, and opportunities for successful knowledge transfer that fit 
older worker learning preferences and attitudes toward learning. 

Ford and Orel (2005) affirm the importance of biological changes, changes in sensory systems, and 
cognitive changes as elements that inform design decisions when establishing training for older workers. Moreover, 
older workers are less focused on learning for career advancement than for empowerment and self-actualization. 
Training considerations for older workers should include (a) environmental factors, such as proper and balanced 
lighting, modifications to computers (e.g., trackball mice, large-screen monitors) to facilitate operations, and 
bathrooms in convenient locations, (b) task factors, such as avoidance of timed tests, strategies that include audio 
and visual aids, and frequent summarization, and (c) personal factors, such as including older workers in the 
planning and implementation of their learning experiences, and a nurturing learning environment that makes older 
workers feel valued and reduces anxiety about new learning situations. Opportunities should include training to 
update critical skills to help older workers perform in their current positions, and skill development training to 
prepare older employees for new jobs or functions. This enables older workers to be redeployed and/or retained in 
the event of redundancies due to restructuring or other organizational changes (Koc-Mendard, 2009). 

Newell, Arnott, Carmichael, and Morgan (2007) offer some specific strategies for involving older adults in 
the design process based on a theater model of software usability testing. In lieu of traditional usability testing 
methods such as laboratories with two-way mirrors to observe human-software system interactions, the authors 
utilized theater professionals (actors, script writers, directors) to produce a series of short dramatized scenarios to 
create a shared context for discussions between potential users of a software package and the designers during the 
requirements-gathering phase. This enables the designers to develop an awareness of the issues and challenges of 
older adults working with new software, while enabling older adults to “see themselves” in the scenarios, so that 
they could then better articulate their needs. 

An alternative stream of thought holds that adherence to instructional design processes is by definition 
inclusive. Regardless of which instructional design model is used, the needs/characteristics of learners, as well as the 
context (organizational, individual) in which the learning takes place, will be taken into account. Consequently, 
instructional designers will automatically become aware of older worker needs, motivation, and self-efficacy and 
will use instructional strategies that move learners from simple to complex tasks with continuous positive feedback 
and emphasis on relevance of the content to work (Rothwell, Sterns, Spokus, & Reaser, 2008). Fisk, Rogers, 
Charness, Czaja, and Sharit (2009) contend that all products - including software applications - should be 
developed using the principles of user-centered design (UCD) that includes: Task analysis; empirical measurement; 
iterative design and testing, and; integrated design with all aspects of the usability design process working in parallel 
and under the coordination of a single person. When developing computerized instruction, the principles of general 
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instructional and multimedia design inform the choice of strategies that can compensate for the cognitive and 
physical effects of normal aging (Van Gerven, Paas, and Tabbers, 2006). For example, to compensate for older 
workers’ reduced mental processing capacity, content presentation should be bimodal (audio-visual), with worked 
examples in lieu of practice problems, and redundant information omitted. 

Training Performance Outcomes 

The second broad issue area in the literature focuses on how older workers perform in training situations. In 
a meta-analysis of 41 empirical studies published from 1950 to 2000, Callahan, Kiker, and Cross (2003) sought to 
compare the effectiveness (as measured by post-training skills tests) of three instructional methods - lecture, 
behavior modeling/demonstration, and activity-based learning - with four instructional support mechanisms, namely 
supplementary materials, training pace, learner feedback, and small group size. Focusing only on studies that 
examined these relationships empirically and that contained a pretest-postest study design with reported effect sizes, 
the authors found that training in small groups or training that enabled older learners to progress at their own pace 
was associated with higher levels of observed training performance relative to large-group or paced training. 

Empirical studies of work-related training performance outcomes conducted since Callahan, Kiker and 
Cross’ (2003) review have been limited in number but varied in scope, design and sample characteristics. In 
recognition of the importance of technology to the workplace, some studies have focused on which instructional 
strategies and methods that incorporate technology best enable older workers to advance their skills. In a study of 67 
adults 40-80 years of age, Chillarege, Nordstron, and Williams (2003) sought to compare the effectiveness of two 
different training goals - learning goal vs. job performance goal - and two training strategies - error management 
vs. error avoidance - on older workers’ performance in a Microsoft Word software training course. Study results 
show higher scores on the word processing knowledge post-tests by participants in the learning goals and error 
management cells than those in the performance goals/error avoidance cells. The authors conclude that these results 
are consistent with other studies that show older workers learn best in situations where mistakes are encouraged 
rather discouraged or penalized. 

As part of a research program that focuses on older adults and computer-based work, Sharit, Czaja, 
Hernandez, et al. (2004) evaluated the ability of 52 adults 50 and older to perform an e-mail-based information 
search and retrieval customer service task for a fictitious Internet-based company. Study findings showed significant 
differences by age on only two variables (total e-mails sent, search time) out of seven, with participants over 65 and 
older similar to their 50-64 year old counterparts in demonstrating progressive improvement on each of the task 
performance measures. Task performance results, in combination with qualitative exit interview data, affirned the 
ability of older participants to learn tasks reflective of the technically oriented work environment. The authors also 
conclude that the results affirm the importance of providing older participants with the skills and confidence in using 
computer-interactive tools, so that cognitive resources can immediately be directed to developing a mental model of 
the task rather than to concerns related to technology use. 

Using the National Institute on Aging (NIA) and the National Library of Medicine (NLM) guidelines for 
designing information technology for older adults, Stolz-Loike, Morrell, and Loike (2005) conducted usability 
testing of a computer-based e-learning software package designed to teach technology skills and business 
information to adults 50-69 who are still in the workforce. The NIA/NLM design guidelines are intended to address 
the cognitive and perceptual aspects of the aging process and cover text features, color, graphics, site design, 
navigation and content delivery. Test results from the 20 participants affirmed the overall usability of the software 
package. However, participants experienced some difficulty in completing tasks that required multiple steps. The 
authors conclude that although the study supports the ability of mature adults to use technology-based materials 
when they are custom-designed for their needs, the study does not address either engagement with or adoption of the 
those materials. 

Wallen and Mulloy (2006) sought to test the impact of age-related cognitive declines by comparing the 
training outcomes of 50 factory workers ranging in age from 26 to 64 on computer-based safety training modules. 
Following a test of prior knowledge in the subject area, participants were randomly assigned to one of three versions 
of the module: Text only, text with pictures, and text with pictures and audio. Upon completing the training module, 
participants were given a multiple choice test to assess their recall of the module content, and then a problem- 
solving test to measure knowledge transfer. Study results indicated no significant differences between workers over 
45 and those under 45 in module content recall. However, there were significant differences by age on the 
knowledge transfer test, with younger workers outperforming their older colleagues. The gap between younger and 
older workers was not as pronounced among participants assigned to the multimedia training module. The authors 
conclude that multimedia computer-based training is more effective for training a multigenerational workforce. 
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because older workers are at less of a disadvantage versus younger workers when trained with multiple modes of 
presentation. 

The effectiveness of specific technology-based instructional strategies has also been examined in cross- 
national contexts. For example, in developing a framework for training workers 40 years of age and older that are 
employed in the Information and Communications Technologies (ICT) sectors in the European Union, 
Themistocleous, Koumaditis, Mantzana, and Morabito (2010) compare participant performance in an online course 
with a fixed schedule and voluntary use of discussion forums, to the same course offered with a flexible, self-paced 
schedule and discussion forum use embedded in group work requirements. Based on higher course completion rates 
and forum participation in the self-paced design, the authors conclude that older ICT workers can learn in a flexible, 
interactive online setting that combines collaboration with learning-by-doing. 

Other empirical studies on the training performance of older workers start from the assumption that good 
training design for older workers applies to both work and nonwork settings. These studies are grounded in the 
stream of gerontology that focuses on the cognitive capacities of adults across the lifecycle. For example, Jamieson 
and Rogers (2000) sought to compare the effectiveness of blocked practice schedules, where the learner practices a 
sequence of tasks repeatedly and in a fixed order, versus random practice schedules, where the learner practices a 
sequence of tasks non-consecutively. Using a simulated automatic teller machine (ATM) as the technology to be 
learned, 80 adults age 18-80 were randomly assigned to the blocked and random practice groups respectively and 
given a series of tasks to perform (e.g., cash withdrawal, transfers from checking to savings). A key finding was that 
older adults' performance in the random condition was not significantly different from younger adults' performance 
in the blocked condition. The authors infer that random practice is better for all age groups in the transfer of learning 
to novel tasks on a novel ATM. The authors conclude that with specific types of practice, older adults may be able to 
overcome limited computer experience and perform as well as younger adults on a computer-based task. The authors 
caution, however, that even after instruction and 50 trials of practice with an ATM, the older individuals remained 
slower, less accurate in their menu selections, and more likely to forget to take their receipt or their cash. 
Consequently, new technologies will require extensive training and exposure before older adults will be capable of 
using them to their full functionality. 

Sanders, Gonzalez, Murphy, et al. (2002) state that lifelong learning is a means of coping with the changing 
demands of daily living in both work and nonwork situations. Drawing on Schmidt and Bjork’s (1992) findings that 
some training conditions hinder learning by reducing subsequent retention and generalization, Sanders et al. (2002) 
sought to assess whether training content difficulty affects the degree of retention and generalization of a newly 
acquired cognitive skill. The authors compared retention and performance outcomes among 116 adults age 18-80 on 
an algorithm for mentally squaring two-digit numbers under high vs. low content variability conditions, such as 
number of problems used and manipulation of the Constant values. Study results reveal significant age differences in 
the effect of high variability training on generalization rather than on retention. The authors attribute these 
differences to the known age differences in working memory capacity and learning efficiency between younger and 
older adults. They conclude that regardless of context - work-related or nonwork related - training should be 
adjusted to the capabilities of the older learner to accommodate reduced working memory and processing speed. 

Using data from a previously conducted simulation of a customer support task (Sharit, et al., 2004), Nair, 
Czaja, and Sharit (2007) examined the role of age, cognitive abilities, prior experience, and knowledge in skill 
acquisition on task performance among 52 older adults over four consecutive days of training. Study results 
indicated that age is the strongest predictor of task performance, with increased age associated with lower initial 
performance. However, results also indicated that the rates of change in performance over time is higher for older 
adults than for younger adults, affirming a previous conclusion that older workers need to be provided with both 
practice and memory aids. 

Ownby, Czaja, Loewenstein, and Rubert (2008) sought to identify the specific cognitive abilities that are 
related to older adults successfully completing training for computer-related tasks. A total of 41 7 participants age 
20-75 were divided into three groups, with each group assigned different data entry, database inquiry, and account 
balancing tasks designed to simulate real-world customer service tasks. Day 1 of the 5-day training included a series 
of cognitive and computer skill assessments, with task training starting on Day 2 and practice tasks on Days 3 to 5. 
Study results indicated that age, computer experience, and measures of short-term memory and psychomotor speed 
predicted older participants successful completion of training. However, the lack of a significant interaction between 
age as a stand-alone variable and training success led the authors to infer that age-related differences in variables 
such as psychomotor speed are more important to understanding older workers’ training success than just age alone. 
The authors conclude that age-related changes in cognitive abilities should be taken into account when designing 
training on job-related tasks. 
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Discussion 



The results of the review, graphically illustrated in Figure 1, raise some challenges for professionals 
seeking to guide their training staff in designing interventions inclusive of both younger and older workers. One 
challenge concerns the lack of a clear, consistent definition of “older”. The U.S. Age Discrimination in Employment 
Act (ADEA) of 1967 defines an older worker as 40 years of age and older (U.S. Employment Equal Opportunity 
Commission, n.d.). Chillarege, Nordstron, and Williams (2003), and Themistocleous, Koumaditis, Mantzana, and 
Morabito (2010) adhere to this definition. However, the other 15 studies in this review each use different defintions, 
ranging from 50+ to 65+. 




Figure 1. Literature themes on age-inclusive work-related training design 

These definition inconsistencies reflect a definition dilemma that pervades the training literature as a whole, 
with as many definitions of “older” as there are topics linked to older worker training. For example, definitions in 
the workforce training literature begin as low as 35 (Jacobson, Lalonde & Sullivan, 2005) and 39 (Delgoulet & 
Marquie, 2002), while other studies deem age 50 to be the cutoff point (Simpson, Greller & Stroh, 2002; Greller, 
2006; Armstrong-Stassen & Schlosser, 2007). In a review of 70 studies published after 1990 that address older 
adults’ computer learning and usage, Kim (2008) found that studies dealing with age-related differences define 
“older” as 65+. 

Another challenge posed by the studies in this review concerns the treatment of older workers as a 
monolithic group, with age-related generalizations about cognition and learning capacity as the basis for training 
design decisions. The studies in this review focus on mapping the cognitive and physical declines associated with 
the normal aging process to instructional strategies such as self-paced learning, learning in small groups with 
generational peers, and with feedback for reinforcement. However, there is some disagreement in the literature as to 
the impact of cognitive and physiological aging on the ability to learn. Charness, Czaja, and Sharit (2007) note that 
people age differently, making group averages unreliable. Further, researchers from a variety of disciplines have 
stated that older adults consist of multiple segments, with a variety of needs, preferences, capabilities, and life 
experiences (Bouvier & De Vita, 1991; Reisenwitz & Iyer, 2007; Lipschultz, Hilt, & Reilly, 2007; Moseley & 
Dessinger, 2007). Level of education and the position of the older worker in his/her organization are also critical 
differentiators (Schaie, 2005; Schmidt, 2007). 

A third challenge is the role of context and content. Context refers to the interaction of situational factors in 
which training takes place, and which need to be identified and accommodated to inspire learning and knowledge 
transfer (Tessmer & Richey, 1997). Content reers to what is to be learned. As noted by Choi and Hannafin (1995), 
context provides the framework for learning, but content determines its authenticity and veracity. The ability to 
select and use a variety of techniques for determining training context and content are core competencies of the 
design professional (International Board of Standards for Training, Performance, and Instruction, 2000). 

Only one of the empirical studies in this review addressed training in a real-world organization with job- 
specific training content and desired training outcomes (Wallen & Mulloy, 2006). The other empirical studies utilize 
quasi-experimental designs in laboratory-like settings -some providing financial incentives to study participants - 
testing tasks unrelated to skills/competencies required on the job or to solving real-world performance problems. As 
such, it is difficult to extrapolate those results to real-world training settings, where workers have a real stake and 
where the risks/rewards of training participation and success may directly impact their work situation. 
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The importance of connecting training activities and events with the real world of work is grounded in the 
situated learning theory of knowledge acquisition (Lave & Wenger, 1990). Situated learning posits that learning 
needs to be presented in an authentic context, with settings and applications that would normally involve that 
knowledge. Situated learning serves as the model for skills and competencies training in a number of industries, 
including healthcare (Woolley & Jarvis, 2007), information technology (Sharma & Yetton, 2007), and engineering 
(Sense, 2007). 

Moving Towards Multigenerational Training Design 

Although this review presents a number of challenges associated with the literature on multigenerational 
workforce training design, the review does have limitations. The conceptual and empirical studies included in this 
review employ a variety of audience definitions, methodologies, and focus on different aspects of training design for 
older workers, all examined through the lens of systematic instructional design. As such, a different exploratory lens 
may lead to a different selection of studies for review. Nevertheless, this review enables researchers and training and 
development practitioners alternative insights into the challenges of multigenerational workforce training design. It 
also suggests some opportunity avenues for addressing those challenges. One avenue is sharing real-world case 
studies of organizations engaging in age-inclusive training design. Case studies, such as the one describing the 
application of instructional design processes at the U.S. National Aeronautics and Space Administration (NASA), 
have helped HR practitioners make the connection between best practices and methods and models identified in the 
research literature (Brock, 2009). 

Another avenue lies in the evolving literature on training and productivity. In the U.S. manufacturing 
sector, for example, cost-benefit analyses have shown the relationship between training and employee productivity 
(Sepulveda, 2009). Analysis by employee age segments would help to address the basic question of whether or not 
older workers are “worth” the training investment. In short, there is a need for more research grounded in real work- 
related training situations identifying the extent to which training context and content affects design decisions, and 
training and job performance outcomes. The extent to which organizations have actually adapted their instructional 
design procedures and technologies to be inclusive of older workers also deserves further investigation. 

At the moment, multigenerational training design is not quite “there” yet in terms of either research or 
practice. However, pockets of practice are evolving and are being shared. Moreover, there is a growing recognition 
that inclusive training design is part of a larger context focusing on how organizations manage their talent (Lawler, 
2008; Codings & Mellahi, 2009). Continued progress in these areas will offer some concrete direction to those 
professionals charged with developing and implementing effective training strategies for an increasingly 
multigenerational workforce. 
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